Generating neural network based perceptual artifact segmentations in modified portions of a digital image

ABSTRACT

Methods, systems, and non-transitory computer readable storage media are disclosed for generating neural network based perceptual artifact segmentations in synthetic digital image content. The disclosed system utilizing neural networks to detect perceptual artifacts in digital images in connection with generating or modifying digital images. The disclosed system determines a digital image including one or more synthetically modified portions. The disclosed system utilizes an artifact segmentation machine-learning model to detect perceptual artifacts in the synthetically modified portion(s). The artifact segmentation machine-learning model is trained to detect perceptual artifacts based on labeled artifact regions of synthetic training digital images. Additionally, the disclosed system utilizes the artifact segmentation machine-learning model in an iterative inpainting process. The disclosed system utilizes one or more digital image inpainting models to inpaint in a digital image. The disclosed system utilizes the artifact segmentation machine-learning model detect perceptual artifacts in the inpainted portions for additional inpainting iterations.

BACKGROUND

Recent years have seen significant advancements in the fields of digitalimage processing and machine-learning. Many industries utilizemachine-learning techniques to automatically generate and modify digitalimages for a variety of uses such as correcting errors, object removal,or dataset generation/augmentation. For example, some industries providetools for performing digital image inpainting operations to digitallyremove objects from digital images and automatically fill the holecreated by removing the objects utilizing one or more neural networks.Accurately generating or modifying digital images utilizingmachine-learning models can be a difficult and resource-expensive task,particularly for certain types of digital image content. Conventionalimage generation systems are limited in accuracy and flexibility ofoperation by introducing perceptual artifacts into syntheticallygenerated/modified portions of digital images.

SUMMARY

This disclosure describes one or more embodiments of methods,non-transitory computer readable media, and systems that solve theforegoing problems (in addition to providing other benefits) byutilizing neural networks to detect perceptual artifacts in digitalimages in connection with generating or modifying digital images. Thedisclosed systems determine a digital image including one or moresynthetically modified portions, such as a digital image generated viaan image generation neural network or modified utilizing a digital imageinpainting model. The disclosed systems utilize an artifact segmentationmachine-learning model to detect perceptual artifacts in thesynthetically modified portion(s). In one or more embodiments, thedisclosed systems train the artifact segmentation machine-learning modelto detect perceptual artifacts based on labeled artifact regions ofsynthetic training digital images.

In some embodiments, the disclosed systems also utilize the artifactsegmentation machine-learning model in an iterative inpainting process.Specifically, the disclosed systems detect a perceptual artifact in asynthetically modified digital image in a first inpainting iteration anddetermine a first artifact segmentation corresponding to the perceptualartifact. In response to determining the first artifact segmentation,the disclosed systems perform an additional inpainting iteration bygenerating an additional synthetically modified portion for the firstartifact segmentation. Accordingly, the disclosed systems perform aplurality of iterations of an inpainting process to continue inpaintingportions of the digital image and detecting artifacts after eachinpainting step. The disclosed systems thus provide flexible andaccurate detection of perceptual artifacts in synthetically modifieddigital images in digital image editing processes.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments will be described and explained with additionalspecificity and detail through the use of the accompanying drawings inwhich:

FIG. 1 illustrates an example system environment in which an artifactsegmentation system can operate in accordance with one or moreimplementations.

FIG. 2 illustrates a diagram of the artifact segmentation systemdetermining artifact segmentations from a digital image in accordancewith one or more implementations.

FIG. 3 illustrates a diagram of the artifact segmentation systemtraining an artifact segmentation machine-learning model utilizinglabeled artifact regions of synthetic training digital images inaccordance with one or more implementations.

FIG. 4 illustrates a diagram of a process of the artifact segmentationsystem determining a labeled artifact region of a synthetic trainingdigital image in accordance with one or more implementations.

FIG. 5 illustrates digital images in an iterative digital imageinpainting process in accordance with one or more implementations.

FIGS. 6A-6B illustrate diagrams of the artifact segmentation systemperforming an iterative digital image inpainting process in accordancewith one or more implementations.

FIG. 7 illustrates a graph diagram of a comparison of artifact ratiometrics relative to digital image inpainting iterations in accordancewith one or more implementations.

FIGS. 8A-8B illustrate diagrams of the artifact segmentation systemselecting a digital image inpainting model based on performance of aplurality of digital image inpainting models in accordance with one ormore implementations.

FIG. 9 illustrates a diagram of the artifact segmentation systemperforming an iterative digital image inpainting process utilizing aplurality of digital image inpainting models in accordance with one ormore implementations.

FIG. 10 illustrates a diagram of the artifact segmentation systemcomparing performance of image generation neural networks in accordancewith one or more implementations.

FIG. 11 illustrates a graphical user interface for displaying detectedartifact segmentations in a digital image in accordance with one or moreimplementations.

FIG. 12 illustrates a diagram of the artifact segmentation system ofFIG. 1 in accordance with one or more implementations.

FIG. 13 illustrates a flowchart of a series of acts for detectingperceptual artifacts utilizing an artifact segmentation machine-learningmodel in accordance with one or more implementations.

FIG. 14 illustrates a flowchart of a series of acts for utilizing anartifact segmentation machine-learning model to perform iterativedigital image inpainting in accordance with one or more implementations.

FIG. 15 illustrates a block diagram of an exemplary computing device inaccordance with one or more embodiments.

DETAILED DESCRIPTION

This disclosure describes one or more embodiments of an artifactsegmentation system that detects perceptual artifacts in syntheticallymodified digital images. In one or more embodiments, the artifactsegmentation system utilizes an artifact segmentation machine-learningmodel to detect perceptual artifact segments in synthetic portions ofdigital images. In particular, the artifact segmentation system utilizesthe artifact segmentation machine-learning model to detect visuallynoticeable artifacts, such as broken structures or color blobs, insynthetically generated digital image content. The artifact segmentationmachine-learning model includes parameters learned based on user-labeledperceptual artifact regions in synthetic training digital images.Additionally, in some embodiments, the artifact segmentation systemutilizes the artifact segmentation machine-learning model to detectperceptual artifacts for a variety of digital image tasks and neuralnetwork analysis tasks.

In one or more embodiments, as mentioned, the artifact segmentationsystem utilizes an artifact segmentation machine-learning model todetect perceptual artifacts in a digital image. Specifically, theartifact segmentation system determines a digital image that includes adigital image synthetically generated by an image generation neuralnetwork. For example, an image generation neural network (e.g., adigital image inpainting model) generates one or more synthetic portionsfor inserting into a digital image based on an image mask. In additionalembodiments, an image generation neural network generates a newsynthetic digital image.

According to one or more embodiments, the artifact segmentation systemutilizes an artifact segmentation machine-learning model to detectperceptual artifacts within one or more synthetically modified portionsof a digital image. Specifically, the artifact segmentationmachine-learning model determines perceptual artifact regions indicatingartifacts corresponding to pixels in the synthetically modifiedportions. Additionally, the artifact segmentation machine-learning modeldetermines artifact segmentations corresponding to the predictedperceptual artifact regions.

In at least some embodiments, the artifact segmentation machine-learningmodel includes parameters learned based on labeled artifact regions in aplurality of synthetic training digital images. For example, theartifact segmentation system provides training digital images includingsynthetically modified portions for display to labeling devices andcorresponding users to label perceptual artifact regions. The artifactsegmentation system uses the training digital images including thelabeled perceptual artifact regions to update parameters of the artifactsegmentation machine-learning model.

In one or more embodiments, the artifact segmentation system utilizesthe trained artifact segmentation machine-learning model to detectperceptual artifacts of synthetically generated image content for use ina variety of digital image applications. For instance, the artifactsegmentation system utilizes the artifact segmentation machine-learningmodel for digital image tasks, such as determining whether a digitalimage has been modified, whether a modified digital image requiresadditional modifications to remove perceptual artifacts, or forcomparing performance of different image generation neural networks.Additionally, in one or more embodiments, the artifact segmentationsystem generates an artifact ratio metric that indicates a size ratio ofperceptual artifacts in synthetically generated image content inconnection with performing various digital image tasks.

As mentioned, in one or more embodiments, the artifact segmentationsystem leverages perceptual artifact detection to improve imagegeneration tasks. In particular, the artifact segmentation systemutilizes an artifact segmentation machine-learning model to detectperceptual artifacts to perform iterative digital image inpainting. Toillustrate, the artifact segmentation system performs an iterativedigital image inpainting process by iteratively generating syntheticdigital image content and detecting perceptual artifacts within thesynthetically generated digital image content. Accordingly, the artifactsegmentation system performs a plurality of iterations of digital imageinpainting to successively reduce the perceptual artifacts in a digitalimage with each iteration.

According to one or more embodiments, the artifact segmentation systemutilizes the artifact segmentation machine-learning model to determine afirst artifact segmentation within first synthetically modified digitalimage content. In response to determining the first artifactsegmentation, the artifact segmentation system utilizes a digital imageinpainting model to generate second synthetically modified digitalcontent for inserting into the digital image according to the firstartifact segmentation. Additionally, the artifact segmentation systemperforms a plurality of such artifact detection and digital imageinpainting iterations to reduce the perceptual artifacts within thedigital image.

In some embodiments, the artifact segmentation system utilizes aplurality of digital image inpainting models during an iterative digitalimage inpainting process. Specifically, in one or more embodiments, theartifact segmentation system selects a digital image inpainting modelfrom the plurality of digital image inpainting models to use during aparticular inpainting iteration and/or for specific digital imagecontent. Thus, the artifact segmentation system can utilize differentdigital image inpainting models for generating synthetic digital imagecontent during different inpainting iterations. Additionally, in someembodiments, the artifact segmentation system utilizes an artifact ratiometric to select a particular digital image inpainting model.

As mentioned, conventional image generation systems have a number ofshortcomings in relation to flexibility and accuracy of operation. Forexample, some conventional image generation systems utilize imagegeneration neural networks to generate synthetic digital image content.While such conventional image generation systems can perform certaintypes of image content generation tasks with accuracy, theseconventional systems often lack accuracy in digital image inpainting orother reconstruction/restoration operations. Specifically, utilizingconventional image generation systems to generate synthetic digitalimage content for large holes or complex structures within a hole (e.g.,due to object removal in a digital image) can result in significantinpainting artifacts, such as broken/imperfect structures (e.g.,disconnected or distorted lines), color bleeding, or color blobs.Accordingly, fixing such artifacts in conventional systems typicallyrequires manual user corrections or intervention, which can be very timeconsuming even for expert users.

Additionally, as a result of the inaccuracies of conventional imagegeneration systems in certain common use cases, the conventional systemsalso lack flexibility. In particular, because the conventional imagegeneration systems are not able to accurately deal with large holeregions and complex structures, the conventional systems are limited touse in certain digital image editing cases. More specifically, suchconventional systems are limited to use in image editing operationsinvolving foreground reconstruction or for background reconstruction inconnection with small object removal.

Furthermore, many conventional image generation systems utilize imageanalysis metrics that compare generated image content to an originalimage in terms of content/pixel similarity. Although such metrics canprovide accurate pixel comparisons when a ground-truth image isavailable, digital image inpainting processes involving object removaltypically do not have access to a ground-truth image with objectsremoved. Accordingly, conventional systems that rely on such imageanalysis metrics can produce inaccurate results due to poor/inaccuratetraining. Some conventional image generation systems utilizequantitative metrics computed on entire images over large evaluationdatasets. Such conventional systems lack usefulness and accuracy foranalysis of individual hole regions.

The disclosed artifact segmentation system provides a number ofadvantages over conventional systems. For example, the artifactsegmentation system improves the flexibility and accuracy of computingdevices that implement digital image generation and editing. In contrastto conventional systems that introduce significant artifacts intosynthetically generated digital image content via the use ofconventional image generation neural networks, the artifact segmentationsystem reduces perceptual artifacts in synthetically generated digitalcontent via the use of an artifact segmentation machine-learning model.Specifically, by utilizing a machine-learning model trained onuser-labeled perceptual artifacts of digital images, the artifactsegmentation system more accurately detects perceptual artifacts insynthetic content consistent with human perception.

Additionally, the artifact segmentation system provides improvedaccuracy over conventional systems by leveraging a metric based on therelative size of perceptual artifacts in synthetic digital imagecontent. In particular, in contrast to conventional systems that utilizecomparison metrics that rely on having a ground-truth image, theartifact segmentation system determines artifact ratio metrics based onthe sizes of detected artifacts relative to the sizes of the inputholes. By determining the ratio of perceptual artifacts relative to thesynthetically generated content, the artifact segmentation systemprovides an interpretable, intuitive, and simple metric for evaluatingand improving the accuracy of synthetic content generated by imagegeneration neural networks. Furthermore, by generating an artifact ratiometric based on perceptual artifacts detected by an artifactsegmentation machine-learning model, the artifact segmentation systemcan automatically evaluate object removal performance, such as in adigital image inpainting process.

Furthermore, the artifact segmentation system provides improvedflexibility and accuracy over conventional systems by utilizingautomatic perceptual artifact detection in digital image inpaintingprocesses. For example, in contrast to conventional systems that are notcapable of automatic artifact detection and segmentation, the artifactsegmentation system uses machine-learning based detection of perceptualartifacts to provide an iterative inpainting process. Specifically, theartifact segmentation system automatically detects and segmentsperceptual artifacts after each digital inpainting operation todetermine input regions for subsequent digital inpainting operations.This results in greater accuracy by consistently reducing perceptualartifact regions and improving color/structural content for a number ofdifferent digital image inpainting models. Furthermore, by selectingfrom a plurality of different digital image inpainting models for eachdigital inpainting operation, the artifact segmentation system increasesthe types of digital image content to which the artifact segmentationsystem can apply digital inpainting operations.

As illustrated by the foregoing discussion, the present disclosureutilizes a variety of terms to describe features and advantages of theartifact segmentation system. Additional detail is now providedregarding the meaning of such terms. For example, as used herein, theterm “synthetically modified portion” (or “synthetically generatedportion”) refers to a portion of a digital image generated or modified(e.g., utilizing a machine-learning model). To illustrate, asynthetically modified portion includes a portion of a digital imagegenerated by a digital image inpainting model during an image inpaintingprocess. Alternatively, a synthetically modified portion includes aportion of a digital image generated by another type of image generationneural network (or otherwise modified, such as, by a digital imageediting application or algorithm).

As used herein, the term “machine-learning model” refers to one or morecomputer algorithms that can be tuned (e.g., trained) based on inputs toapproximate unknown functions. In particular, a machine-learning modelutilizes algorithms to learn from, and make determinations on, knowndata by analyzing the known data to learn to generate outputs thatreflect patterns and attributes of the known data. For instance, amachine-learning model can include, but is not limited to, one or moreneural network layers, such as a multi-layer perceptron, a convolutionalneural network, a recurrent neural network, a generative adversarialneural network, a feed forward neural network, or any combinationthereof. A machine-learning model can learn high-level abstractions indata to generate data-driven determinations, predictions, or decisionsfrom the known input data. Furthermore, as described herein, in someembodiments, an “image generation neural network”) includes one or moreneural network layers for generating one or more synthetic portions of adigital image. In one or more embodiments, a “digital image inpaintingmodel” includes one or more neural network layers to generate syntheticdigital image content for inserting into one or more portions of anexisting digital image. Additionally, in some embodiments, an “artifactsegmentation machine-learning model” includes one or more neural networklayers to detect perceptual artifacts in synthetically modified portionsof digital images.

As used herein, the term “perceptual artifact” refers to a visible errorin synthetically generated image content. For example, a perceptualartifact includes unexpected structures or colors in syntheticallygenerated image content that given contextual human understanding ofstructures and colors. To illustrate, perceptual artifacts includebroken structures, color blobs, color bleeding, or distorted lines insynthetically generated content. In some embodiments, perceptualartifacts are a result of machine-learning models lacking contextualunderstanding of real-world shapes and objects that humans learn.

Additionally, as used herein, the term “perceptual artifact region” (or“predicted perceptual artifact region”) refers to a portion of a digitalimage that an artifact segmentation machine-learning model predicts toinclude at least one perceptual artifact. Furthermore, as used herein,the term “artifact segmentation” refers to an indication of a portion ofa digital image including a perceptual artifact. To illustrate, anartifact segmentation includes a mask with one or more boundariescorresponding to perceptual artifact regions in a digital image aspredicted by an artifact segmentation machine-learning model.

As used herein, the term “artifact ratio metric” refers to a valueindicating a relative size of one or more artifacts within asynthetically modified portion of a digital image. For instance, anartifact ratio metric includes a ratio of a size of one or morepredicted perceptual artifact regions relative to a size of one or moresynthetically modified portions of a digital image.

As used herein, the term “labeled artifact region” refers to a portionof a digital image marked by a user as including an artifact.Specifically, a labeled artifact region includes a manually identifiedgroup of pixels as including a perceptual artifact within asynthetically modified portion of a digital image. More specifically, alabeled artifact region includes a marked portion of a synthetictraining digital image for training an artifact segmentationmachine-learning model.

As used herein, the term “digital image mask” refers to a mapping ofassigned values to pixels of a digital image for restricting one or moreoperators on the digital image to one or more areas defined by theassigned values. For example, a digital image mask includes zero andnon-zero values to indicate one or more objects of a digital image in adigital image editing process. To illustrate, a digital image maskincludes a “hole mask” indicating a portion of an object removed (or tobe removed) from a digital image and replaced with syntheticallymodified image content in a digital image inpainting process.Additionally, in some embodiments, a digital image mask includes maskingvalues indicating one or more artifact segmentations in one or moresynthetically modified portions of a digital image.

As used herein, the term “inpainting iteration” refers to a digitalimage editing process of replacing at least a portion of a digital imagewith synthetically generated digital image content. Specifically, aninpainting iteration includes determining one or more input regions(e.g., based on a digital image mask) and replacing one or more portionsof the digital image indicated by the one or more input regionsutilizing a digital image inpainting model. In some embodiments, aninpainting iteration includes identifying an artifact in previouslymodified synthetic image content and generating additional syntheticimage content. Accordingly, a plurality of inpainting iterations includesuccessively identifying regions of a digital image to replace withsynthetically generated digital image content.

Turning now to the figures, FIG. 1 includes an embodiment of a systemenvironment 100 in which an artifact segmentation system 102 isimplemented. In particular, the system environment 100 includes serverdevice(s) 104 and a client device 106 in communication via a network108. Moreover, as shown, the server device(s) 104 include a digitalimage system 110, which includes the artifact segmentation system 102.FIG. 1 illustrates that the artifact segmentation system 102 alsoincludes an artifact segmentation machine-learning model 112 and animage generation neural network 114. Additionally, the client device 106includes a digital image application 116, which optionally includes thedigital image system 110 and the artifact segmentation system 102, whichfurther includes the artifact segmentation machine-learning model 112and the image generation neural network 114. In one or more embodiments,as illustrated in FIG. 1 , the system environment 100 also includes adigital image database 118 in communication with the server device(s)104 and/or the client device 106.

As shown in FIG. 1 , the server device(s) 104 includes or host thedigital image system 110. The digital image system 110 include, or bepart of, one or more systems that implement digital image generationand/or digital image editing. For example, the digital image system 110provides tools for viewing, generating, editing, and/or otherwiseinteracting with digital images. To illustrate, the digital image system110 communicates with the client device 106 via the network 108 toprovide the tools for display and interaction via the digital imageapplication 116 at the client device 106.

The digital image system 110 uses the digital images in a variety ofapplications such as databases of digital images (e.g., the digitalimage database 118) or other digital media (e.g., in digital videos). Insome embodiments, the digital image system 110 communicates with theclient device 106 to provide tools for generating or editing digitalimages from the digital image database 118 or for storing in the digitalimage database 118. For instance, the digital image system 110 providestools for generating synthetic digital images for inclusion in atraining dataset at the digital image database 118. The digital imagesystem 110 or another system can utilize the digital images in thedigital image database 118 to train one or more neural networks (e.g.,image generation neural networks or object detection networks).

In some embodiments, the digital image system 110 receives interactiondata for viewing, generating, or editing a digital image from the clientdevice 106, processes the interaction data (e.g., to view, generate, oredit a digital image), and provides the results of the interaction datato the client device 106 for display via the digital image application116 or to a third-party system. Additionally, in some embodiments, thedigital image system 110 receives data from the client device 106 inconnection with editing digital images, including requests to accessdigital images stored at the server device(s) 104 (or at another devicesuch as the digital image database 118) and/or requests to store digitalimages from the client device 106 at the server device(s) 104 (or atanother device).

In connection with generating or editing digital images, the digitalimage system 110 utilizes the artifact segmentation system 102 togenerate synthetic digital image content and detect perceptual artifactsin synthetic digital image content. For example, the artifactsegmentation system 102 utilizes the image generation neural networks114 to generate synthetic digital image content for creating a newsynthetic digital image or for modifying an existing digital image.Additionally, the artifact segmentation system 102 utilizes the artifactsegmentation machine-learning model 112 to detect perceptual artifactsin the synthetically generated digital image content. In one or moreembodiments, the artifact segmentation system 102 utilizes the imagegeneration neural networks 114 and the artifact segmentationmachine-learning model 112 in an iterative digital image inpaintingprocess.

In one or more embodiments, in response to utilizing the artifactsegmentation system 102 to generate synthetic digital image content anddetect perceptual artifacts, the digital image system 110 provides theresulting digital image and/or artifact segmentations to the clientdevice 106 for display. For instance, the digital image system 110 sendsa modified digital image and/or indications of artifact segmentations tothe client device 106 via the network 108 for display via the digitalimage application 116. Additionally, in some embodiments, the clientdevice 106 receives additional inputs to apply additional changes to thedigital image (e.g., based on additional inputs to further modify one ormore portions of the digital image). The client device 106 sends arequest to apply the additional changes to the digital image to thedigital image system 110, and the digital image system 110 utilizes theartifact segmentation system 102 to update the digital image.

In one or more embodiments, the server device(s) 104 include a varietyof computing devices, including those described below with reference toFIG. 15 . For example, the server device(s) 104 includes one or moreservers for storing and processing data associated with digital images.In some embodiments, the server device(s) 104 also include a pluralityof computing devices in communication with each other, such as in adistributed storage environment. In some embodiments, the serverdevice(s) 104 include a content server. The server device(s) 104 alsooptionally includes an application server, a communication server, aweb-hosting server, a social networking server, a digital contentcampaign server, or a digital communication management server.

In addition, as shown in FIG. 1 , the system environment 100 includesthe client device 106. In one or more embodiments, the client device 106includes, but is not limited to, a mobile device (e.g., smartphone ortablet), a laptop, a desktop, including those explained below withreference to FIG. 15 . Furthermore, although not shown in FIG. 1 , theclient device 106 can be operated by a user (e.g., a user included in,or associated with, the system environment 100) to perform a variety offunctions. In particular, the client device 106 performs functions suchas, but not limited to, accessing, viewing, and interacting with avariety of digital content (e.g., digital images). In some embodiments,the client device 106 also performs functions for generating, capturing,or accessing data to provide to the digital image system 110 and theartifact segmentation system 102 in connection with generating orediting digital images. For example, the client device 106 communicateswith the server device(s) 104 via the network 108 to provide information(e.g., user interactions) associated with digital images and artifactsegmentations. Although FIG. 1 illustrates the system environment 100with a single client device, in some embodiments, the system environment100 includes a different number of client devices.

Additionally, as shown in FIG. 1 , the system environment 100 includesthe network 108. The network 108 enables communication betweencomponents of the system environment 100. In one or more embodiments,the network 108 may include the Internet or World Wide Web.Additionally, the network 108 can include various types of networks thatuse various communication technology and protocols, such as a corporateintranet, a virtual private network (VPN), a local area network (LAN), awireless local network (WLAN), a cellular network, a wide area network(WAN), a metropolitan area network (MAN), or a combination of two ormore such networks. Indeed, the server device(s) 104 and the clientdevice 106 communicates via the network using one or more communicationplatforms and technologies suitable for transporting data and/orcommunication signals, including any known communication technologies,devices, media, and protocols supportive of data communications,examples of which are described with reference to FIG. 15 .

Although FIG. 1 illustrates the server device(s) 104 and the clientdevice 106 communicating via the network 108, in alternativeembodiments, the various components of the system environment 100communicate and/or interact via other methods (e.g., the serverdevice(s) 104 and the client device 106 can communicate directly).Furthermore, although FIG. 1 illustrates the artifact segmentationsystem 102 being implemented by a particular component and/or devicewithin the system environment 100, the artifact segmentation system 102can be implemented, in whole or in part, by other computing devicesand/or components in the system environment 100 (e.g., the client device106).

In particular, in some implementations, the artifact segmentation system102 on the server device(s) 104 supports the artifact segmentationsystem 102 on the client device 106. For instance, the server device(s)104 generates or obtains the artifact segmentation system 102 (includingthe artifact segmentation machine-learning model 112 and the imagegeneration neural networks 114) for the client device 106. The serverdevice(s) 104 trains and provides the artifact segmentation system 102and the artifact segmentation machine-learning model 112 and imagegeneration neural networks 114 to the client device 106 for performing adigital image generation/editing process at the client device 106. Inother words, the client device 106 obtains (e.g., downloads) theartifact segmentation system 102 and the artifact segmentationmachine-learning model 112 and image generation neural networks 114 fromthe server device(s) 104. At this point, the client device 106 is ableto utilize the artifact segmentation system 102 (with the artifactsegmentation machine-learning model 112 and the image generation neuralnetworks 114) to generate and/or edit digital images with perceptualartifact detection independently from the server device(s) 104.

In alternative embodiments, the artifact segmentation system 102includes a web hosting application that allows the client device 106 tointeract with content and services hosted on the server device(s) 104.To illustrate, in one or more implementations, the client device 106accesses a web page supported by the server device(s) 104. The clientdevice 106 provides input to the server device(s) 104 to perform meshmapping and/or mesh generation operations, and, in response, theartifact segmentation system 102 or the digital image system 110 on theserver device(s) 104 performs operations to generate and/or edit digitalimages. The server device(s) 104 provide the output or results of theoperations to the client device 106.

As mentioned, the artifact segmentation system 102 detects perceptualartifacts in synthetically generated digital image content. FIG. 2illustrates an overview of the artifact segmentation system 102detecting perceptual artifacts in a digital image. Specifically, FIG. 2illustrates that the artifact segmentation system 102 utilizes anartifact segmentation machine-learning model 112 to detect perceptualartifacts in synthetically modified portions of a digital image.

In one or more embodiments, as illustrated in FIG. 2 , the artifactsegmentation system 102 determines a digital image 200 that includes oneor more synthetically modified portions. For example, the digital image200 includes a photograph of a real-world scene with at least a portionincluding synthetically modified image content. To illustrate, thedigital image 200 includes a synthetically modified portion 202resulting from a digital image inpainting process that includes removinga portion of the digital image 200 and replacing the removed portionwith the synthetically modified portion. In alternative embodiments, thedigital image 200 includes a fully synthetically generated digitalimage.

In one or more embodiments, the artifact segmentation system 102generates the synthetically modified portion 202 of the digital image200 utilizing an image generation neural network 204. Specifically, theimage generation neural network 204 generates the synthetically modifiedportion 202 based on a digital image mask provided for the digital image200. To illustrate, the digital image mask (e.g., a hole mask) includesone or more hole regions indicating one or more portions of the digitalimage 200 to fill/replace with synthetic digital image content, such asin an object removal process (e.g., to remove a foreground object). Theartifact segmentation system 102 utilizes the image generation neuralnetwork 204 to generate synthetic digital image content and modify thedigital image 200 with the synthetically modified portion 202. In one ormore embodiments, the image generation neural network 204 includes adigital image inpainting model or other generative adversarial neuralnetwork for generating synthetic digital image content.

According to one or more embodiments, the artifact segmentation system102 utilizes the artifact segmentation machine-learning model 112 todetect perceptual artifacts in the digital image 200. For instance, theartifact segmentation system 102 utilizes the artifact segmentationmachine-learning model 112 to determine one or more predicted perceptualartifact regions indicating one or more artifacts in the digital image200. To illustrate, as shown in FIG. 2 , the artifact segmentationmachine-learning model 112 processes the digital image 200 to detectperceptual artifacts in the synthetically modified portion 202 of thedigital image 200. For example, the artifact segmentationmachine-learning model 112 determines a predicted perceptual artifactregion 206 based on the synthetically modified portion 202. In one ormore additional embodiments, the artifact segmentation system 102utilizes the artifact segmentation machine-learning model 112 togenerate a plurality of predicted perceptual artifact regionscorresponding to different perceptual artifact in the syntheticallymodified portion 202.

In one or more embodiments, a predicted perceptual artifact regioncorresponds to a plurality of pixels of a perceptual artifact in adigital image. To illustrate, the artifact segmentation machine-learningmodel 112 generates predictions of perceptual artifact regionscorresponding to pixels in the digital image 200. More specifically, asdescribed in more detail below with respect to FIGS. 3-4 , the artifactsegmentation system 102 utilizes an artifact segmentationmachine-learning model 112 trained on a set of training digital imagesthat include user-labeled regions including perceptual artifacts. Thus,the artifact segmentation system 102 utilizes the artifact segmentationmachine-learning model 112 to detect artifacts noticeable to humansbased on contextual human understanding of structures and colors.

Furthermore, in one or more embodiments, the artifact segmentationsystem 102 generates (e.g., utilizing the artifact segmentationmachine-learning model 112) one or more artifact segmentations for adigital image via a digital image mask. In particular, the artifactsegmentation system 102 determines, for each predicted perceptualartifact region in a digital image, an artifact segmentation based onpixels corresponding to the predicted perceptual artifact region. Theartifact segmentation system 102 stores the artifact segmentation in adigital image mask by assigning values to pixels in the digital imagemask according to whether each pixel is inside or outside a boundary ofthe artifact segmentation. Accordingly, the artifact segmentation system102 generates a digital image mask including any number of “holes”corresponding to predicted perceptual artifact regions (e.g., a firsthole corresponding to a first predicted perceptual artifact region and asecond hole corresponding to a second predicted perceptual artifactregion).

As previously mentioned, the artifact segmentation system 102 utilizesan artifact segmentation machine-learning model trained to detectperceptual artifacts based on human-labeled training data. Inparticular, in one or more embodiments, the artifact segmentation system102 determines a training dataset of digital images including syntheticdigital image content. Additionally, the artifact segmentation system102 prepares and provides the training dataset for labeling ofperceptual artifacts by a plurality of users. FIG. 3 illustrates anoverview of the artifact segmentation system 102 preparing synthetictraining digital images for obtaining labeled perceptual artifacts. FIG.3 also illustrates the artifact segmentation system 102 utilizing thelabeled perceptual artifacts to train the artifact segmentationmachine-learning model 112 to detect perceptual artifacts.

According to one or more embodiments, the artifact segmentation system102 determines a synthetic training digital image 300 including asynthetically modified portion 302. For example, as previouslydescribed, the artifact segmentation system 102 (or another system)modified the synthetic training digital image 300 to include thesynthetically modified portion 302 in connection with an object removalprocess. To illustrate, the artifact segmentation system 102 (or anothersystem) utilizes an image generation neural network to remove an objectfrom the foreground of the synthetic training digital image 300 andreplace the object with synthetic digital image content based on aninitial hole mask. Alternatively, the artifact segmentation system 102(or another system) utilizes an image generation neural network tocorrect an imperfection in a digital image (e.g., due to scanning errorsor physical imperfections on a photograph).

As illustrated in FIG. 3 , the artifact segmentation system 102 preparesthe synthetic training digital image 300 to provide to a plurality ofclient devices for labeling. In particular, in one or more embodiments,the artifact segmentation system 102 prepares the synthetic trainingdigital image 300 for presentation on display devices of the clientdevices by dilating an initial hole mask corresponding to thesynthetically modified portion 302. For example, the artifactsegmentation system 102 dilates the initial hole mask or a boundary ofthe synthetically modified portion 302 by a predetermined amount. Toillustrate, the artifact segmentation system 102 dilates the initialhole mask by a predetermined number of pixels, a predetermined ratio ofpixels relative to the size of the initial hole mask and/or the size ofthe digital image.

In additional embodiments, the artifact segmentation system 102generates a visible shape based on the initial hole mask. For instance,in response to determining the initial hole mask, the artifactsegmentation system 102 determines a rectangle that encloses thesynthetically modified portion 302. Specifically, the artifactsegmentation system 102 dilates the initial hole mask by a predeterminedamount and generates a rectangle based on the dilated initial hole mask.As FIG. 3 illustrates, the artifact segmentation system 102 generates arectangle 304 that encloses the synthetically modified portion 302 ofthe synthetic training digital image 300. In some embodiments, theartifact segmentation system 102 generates the rectangle 304 to includeedges at the boundaries of the dilated initial hole mask.

In alternative embodiments, the artifact segmentation system 102 expandsthe rectangle 304 beyond the edges of the dilated initial hole mask. Forexample, the artifact segmentation system 102 expands one or more edgesof the rectangle 304 by a predetermined amount. In additional examples,the artifact segmentation system 102 expands one or more edges of therectangle 304 by a random amount (e.g., a random number of pixels withina range of pixels). In some examples, the artifact segmentation system102 expands one or more edges of the rectangle 304 to one or more edgesof the synthetic training digital image 300 in response to determiningthat the dilated initial mask region is within a threshold number ofpixels of the one or more edges of the synthetic training digital image300.

In response to generating a dilated portion of the synthetic trainingdigital image 300 (e.g., the rectangle 304), the artifact segmentationsystem 102 provides the synthetic training digital image 300 with thedilated portion to a plurality of client devices. Specifically, theartifact segmentation system 102 provides the synthetic training digitalimage 300 to client devices of a plurality of users for manual labelingof perceptual artifacts. For example, the artifact segmentation system102 sends the synthetic training digital image 300 with a plurality ofadditional synthetic training digital images (with dilated indicators)for client devices and corresponding users to label perceptual artifactson each of the digital images (e.g., based on user interaction with theperceptual artifacts).

In one or more embodiments, by dilating the synthetically modifiedportion 302 of the synthetic training digital image 300 for providing toclient devices and corresponding users for labeling of perceptualartifacts, the artifact segmentation system 102 provides the synthetictraining digital images without explicitly displaying the syntheticallymodified portions. More specifically, the artifact segmentation system102 provides the synthetic training digital images without introducingbias into the labels. For instance, by providing the general regions ofthe synthetic training digital images for display at the client devices,the artifact segmentation system 102 prevents bias toward perceptualartifacts. Instead, presenting the general regions allows users toindividually judge the locations of perceptual artifacts in digitalimage content.

In one or more embodiments, the artifact segmentation system 102 alsoprovides an additional version of the synthetic training digital image300 to the client devices. For instance, the artifact segmentationsystem 102 provides an unmarked copy/duplicate of the synthetic trainingdigital image 300 to the client devices for display with the synthetictraining digital image 300. To illustrate, by displaying two copies ofthe synthetic training digital image 300 the artifact segmentationsystem 102 provides a first digital image for labeling and a seconddigital image as a reference. Thus, a user can interact with the clientdevice utilizing a stylus or other input device to label one or moreartifact regions.

According to one or more embodiments, the artifact segmentation system102 obtains a plurality of labeled images from a plurality of clientdevices. In particular, as illustrated in FIG. 3 , the artifactsegmentation system 102 receives a labeled training digital image 306including a labeled artifact region 308 indicating one or moreperceptual artifacts as marked by a user of a client device. Toillustrate, the labeled artifact region 308 includes a portion of thelabeled training digital image 306 marked via a stylus or other inputtool at the client device. In additional embodiments, the artifactsegmentation system 102 receives a plurality of labeled training digitalimages with labeled artifact regions from a plurality of client devices.

In connection with determining labeled artifact regions of syntheticdigital images based on user interaction, the artifact segmentationsystem 102 also utilizes the artifact segmentation machine-learningmodel 112 to generate predicted perceptual artifact segmentations fordigital images. For example, as illustrated in FIG. 3 , the artifactsegmentation system 102 utilizes the artifact segmentationmachine-learning model 112 to generate a predicted perceptual artifactregion 310 based on the synthetically modified portion 302 of thesynthetic training digital image 300. As illustrated, the artifactsegmentation system 102 generates the predicted perceptual artifactregion 310 by labeling a region corresponding to a plurality of pixelsof the synthetic training digital image 300 as including a perceptualartifact.

In one or more embodiments, the artifact segmentation system 102utilizes the labeled artifact region 308 and the predicted perceptualartifact region 310 to train the artifact segmentation machine-learningmodel 112. For instance, the artifact segmentation system 102 determinesa loss 312 based on a difference between the predicted perceptualartifact region 310 and the labeled artifact region 308. To illustrate,the artifact segmentation system 102 determines the loss 312 accordingto the pixel differences between the predicted perceptual artifactregion 310 and the labeled artifact region 308.

The artifact segmentation system 102 then trains the artifactsegmentation machine-learning model 112 based on the loss 312. Inparticular, the artifact segmentation system 102 utilizes the loss 312to update parameters of the artifact segmentation machine-learning model112 to cause the predicted perceptual artifact region 310 to be closerto the labeled artifact region 308. In additional embodiments, theartifact segmentation system 102 performs additional training iterationsby generating updated predicted perceptual artifact regions and updatedlosses to further train the artifact segmentation machine-learning model112. Furthermore, in one or more embodiments, the artifact segmentationsystem 102 generates the loss 312 based on a plurality of labeledartifact regions based on the synthetic training digital image 300and/or a plurality of labeled artifact regions based on a plurality ofsynthetic training digital images.

Additionally, in some embodiments, the artifact segmentation system 102modifies one or more labeled regions of labeled training digital imagesto correct for mislabeled areas. Specifically, FIG. 4 illustrates thatthe artifact segmentation system 102 modifies labeled artifact regionsaccording to synthetically modified portions of digital images. Forinstance, the artifact segmentation system 102 determines a final labelfor a region based on whether artifacts are possible (e.g., due tosynthetic digital image content) within the marked portions of a digitalimage.

In one or more embodiments, as illustrated in FIG. 4 , the artifactsegmentation system 102 determines a synthetic digital image 400 thatincludes a synthetically modified portion 402. For example, thesynthetic digital image 400 includes the synthetically modified portion402 based on an initial hole mask, or other mask associated withgenerating synthetic digital image content. The artifact segmentationsystem 102 also determines a labeled digital image 404 that includes alabeled artifact region 406. To illustrate, the labeled digital image404 includes the labeled artifact region 406 based on user input markinga portion of the labeled digital image 404 at a separate client device.

According to one or more embodiments, the artifact segmentation system102 determines an intersection based on the synthetically modifiedportion 402 and the labeled artifact region 406. In particular, theartifact segmentation system 102 determines a hole mask (e.g., aninitial hole mask) corresponding to the synthetically modified portion402. The artifact segmentation system 102 determines an intersection ofthe hole mask and the labeled artifact region 406 by determining aplurality of pixels from the labeled artifact region 406 that arelocated within a boundary of the hole mask (e.g., based on coordinatesof the pixels corresponding to the hole mask and coordinates of thepixels corresponding to the labeled artifact region 406). In response todetermining the intersection of the hole mask and the labeled artifactregion 406, the artifact segmentation system 102 generates a synthetictraining digital image 408 including a final labeled artifact region 410based on the intersection.

In one or more additional embodiments, the artifact segmentation system102 standardizes labeled artifact regions in synthetic training digitalimages. Specifically, due to the subjective nature of human perception,particularly with regard to perceptual artifacts in digital images, theartifact segmentation system 102 performs one or more verification stepsfor standardizing the labeled artifact regions. For instance, theartifact segmentation system 102 provides the labeled synthetic digitalimages to one or more additional users, such as a set of professionalimage users to cross check the labeled artifact regions of a givendigital image and add or remove portions of the labeled artifactregions. Additionally, the artifact segmentation system 102 provides thelabeled synthetic digital images to one or more additional users (e.g.,an expert image user). By providing the labeled synthetic digital imagesto additional users, the artifact segmentation system 102 provides aprocess for correcting erroneous labels.

As previously mentioned, the artifact segmentation system 102 canutilize the artifact segmentation machine-learning model 112 to performautomatic detection of perceptual artifacts in a variety of digitalimage editing processes. According to one or more embodiments, theartifact segmentation system 102 utilizes the artifact segmentationmachine-learning model 112 to automatically detect perceptual artifactsin an iterative inpainting process. For example, FIG. 5 illustrates aplurality of digital images in a digital image inpainting process.

In one or more embodiments, a digital image 500 is associated with ahole mask 502 corresponding to an object in a foreground of the digitalimage 500. As illustrated in FIG. 5 , the hole mask 502 is overlaid onthe digital image 500 based on a coordinate system of the digital image500 and a similar coordinate system of the hole mask 502. In one or moreembodiments, the hole mask 502 includes a hole region comprising pixelsrepresenting the object in the digital image 500. In additionalembodiments, the hole mask 502 includes a buffer region surrounding theobject.

According to one or more embodiments, the artifact segmentation system102 or another system generates synthetic digital image content toreplace the portion of the digital image 500 corresponding to the holeregion of the hole mask 502. For example, the artifact segmentationsystem 102 utilizes a digital image inpainting model to generatesynthetic digital image content to fill the hole region indicated by thehole mask 502. To illustrate, the artifact segmentation system 102utilizes the digital image inpainting model to replace the object withsynthetic digital image content by attempting to recreate the backgroundbehind the object. Thus, the artifact segmentation system 102 utilizesthe digital image inpainting model to replace the object with asynthetically modified portion.

As illustrated in FIG. 5 , the artifact segmentation system 102generates a first modified digital image 504 including a syntheticallymodified portion in place of the portion of the digital image 500indicated by the hole mask 502. Additionally, in one or moreembodiments, the artifact segmentation system 102 utilizes an artifactsegmentation machine-learning model to automatically detect perceptualartifacts within the synthetically modified portion of the firstmodified digital image 504. For example, the artifact segmentationsystem 102 utilizes the artifact segmentation machine-learning model todetermine a predicted perceptual artifact region and generate anartifact segmentation 506 (e.g., and additional image mask including theartifact segmentation 506) corresponding to the predicted perceptualartifact region.

Furthermore, as illustrated in FIG. 5 , the artifact segmentation system102 performs one or more additional inpainting iterations to generate afinal modified digital image 508. For instance, in response togenerating the artifact segmentation 506, the artifact segmentationsystem 102 utilizes the digital image inpainting model (or a differentdigital image inpainting model) to generate additional synthetic digitalimage content within the portion of the first modified digital image 504corresponding to the artifact segmentation 506. The artifactsegmentation system 102 performs each inpainting iteration by replacingone or more portions of the digital image with synthetical digital imagecontent utilizing one or more digital image inpainting models anddetermining whether the synthetically modified portions includeperceptual artifacts utilizing the artifact segmentationmachine-learning model. As shown in FIG. 5 , the final modified digitalimage 508 is the result of five inpainting iterations, resulting in morerealistic synthetic digital image content than in the first modifieddigital image 504 after a first inpainting iteration.

FIGS. 6A-6B illustrate a plurality of inpainting iterations in a digitalimage inpainting process. Specifically, FIG. 6A illustrates a firstinpainting iteration for modifying a digital image 600 based on aninitial hole mask 602. For example, the artifact segmentation system 102determines the initial hole mask 602 based on user input selecting aregion of the digital image 600. Alternatively, the artifactsegmentation system 102 determines the initial hole mask 602 byutilizing an object detection neural network to identify an object inthe digital image 600 and generate the initial hole mask 602.

In one or more embodiments, the artifact segmentation system 102utilizes a digital image inpainting model 604 a to fill in the portionof the digital image 600 indicated by the initial hole mask 602. Forinstance, as illustrated in FIG. 6A, the artifact segmentation system102 utilizes the digital image inpainting model 604 a to replace theoriginal digital image content (e.g., an object) with a syntheticallymodified portion 606 in the digital image 600. Accordingly, the artifactsegmentation system 102 utilizes the digital image inpainting model 604a to perform object removal in the digital image 600.

In response to generating the synthetically modified portion 606, theartifact segmentation system 102 utilizes the artifact segmentationmachine-learning model 112 to determine whether the syntheticallymodified portion 606 includes perceptual artifacts. As illustrated inFIG. 6A, the artifact segmentation machine-learning model 112 processesthe digital image 600 including the synthetically modified portion 606to detect perceptual artifacts. To illustrate, the artifact segmentationsystem 102 utilizes the artifact segmentation machine-learning model 112to determine a predicted perceptual artifact region 608 within thesynthetically modified portion 606.

Additionally, in one or more embodiments, the artifact segmentationsystem 102 determines to continue performing one or more additionalinpainting iterations based on the predicted perceptual artifact region608. FIG. 6B illustrates a second inpainting iteration in which theartifact segmentation system 102 generates additional syntheticaldigital image content and detects additional perceptual artifacts.Specifically, the artifact segmentation system 102 determines anadditional hole mask 610 including an artifact segmentation based on thepredicted perceptual artifact region 608.

In one or more embodiments, the artifact segmentation system 102utilizes a digital image inpainting model 604 b to generate additionalsynthetic digital image content based on the additional hole mask 610.As illustrated in FIG. 6B, the artifact segmentation system 102 utilizesthe digital image inpainting model 604 b to fill the portion of thedigital image 600 corresponding to the additional hole mask 610 with anadditional synthetically modified portion 612. For example, the artifactsegmentation system 102 utilizes the digital image inpainting model 604b to further refine details and fix errors introduced in thesynthetically modified portion 606 in the first inpainting iteration.

According to one or more embodiments, the digital image inpainting model604 b is the same model as the digital image inpainting model 604 a. Inalternative embodiments, the digital image inpainting model 604 b isdifferent than the digital image inpainting model 604 b. FIGS. 8A—8B and9 and the corresponding description provide additional detail withrespect to utilizing a plurality of digital image inpainting models in adigital image inpainting process.

As illustrated in FIG. 6B, the artifact segmentation system 102 utilizesthe artifact segmentation machine-learning model 112 to determinewhether the additional synthetically modified portion 612 containsperceptual artifacts. In one or more embodiments, the artifactsegmentation machine-learning model 12 processes the digital image 600including the additional synthetically modified portion 612 within thesynthetically modified portion 606. The artifact segmentation system 102utilizes the artifact segmentation machine-learning model 112 todetermine an additional predicted perceptual artifact region 614 withinthe additional synthetically modified portion 612.

In one or more embodiments, the artifact segmentation system 102continues performing inpainting iterations in a digital image inpaintingprocess to reduce perceptual artifacts in the digital image 600. Forinstance, the artifact segmentation system 102 determines a number ofinpainting iterations to perform based on perceptual artifacts detectedin the digital image. Specifically, the artifact segmentation system 102generates an artifact ratio metric that indicates a ratio of thecombined size of perceptual artifacts relative to the size of a digitalimage mask corresponding to a synthetically modified portion.

For example, after each inpainting iteration, the artifact segmentationsystem 102 determines the artifact ratio metric based on the size ofdetected perceptual artifacts relative to the size of an input hole forthe current iteration. To illustrate, the artifact segmentation system102 determines a first artifact ratio metric based on the size of thepredicted perceptual artifact region 608 relative to the size of theinitial hole mask 602. The artifact segmentation system 102 alsodetermines a second artifact ratio metric based on the size of theadditional predicted perceptual artifact region 614 relative to the sizeof the additional hole mask 610.

In one or more embodiments, the artifact segmentation system 102determines whether to perform an additional inpainting iteration basedon an artifact ratio metric of a previous iteration. For instance, theartifact segmentation system 102 compares the artifact ratio metric ofan inpainting iteration to a ratio threshold to determine whether toperform an additional iteration. To illustrate, in response todetermining that the first artifact ratio metric for the first iterationof FIG. 6A meets the ratio threshold (e.g., the combined size of theperceptual artifacts relative to the input hole is at least as high as apredetermined ratio), the artifact segmentation system 102 performs thesecond iteration of FIG. 6B. Additionally, in response to determiningthat the second artifact ratio metric for the second ratio does not meetthe ratio threshold, the artifact segmentation system 102 terminates thedigital image inpainting process and determines a final version of thedigital image.

In additional embodiments, the artifact segmentation system 102determines a number of digital image inpainting operations to performbased on additional considerations. For example, FIG. 7 illustrates agraph diagram 700 comparing the number of inpainting iterations toartifact ratio metrics for different digital image inpainting models. Asshown, performing additional inpainting iterations continually reducesthe artifact ratio metrics for all of the different digital imageinpainting models. For example, as illustrated, performing fiveiterative inpainting operations reduces the artifact ratio metric. Inadditional embodiments, performing additional iterative inpaintingoperations further reduces the artifact ratio metric.

FIG. 7 also illustrates that additional inpainting iterations reducesthe artifact ratio metrics by different amounts for the differentdigital image inpainting models. Accordingly, in one or moreembodiments, the artifact segmentation system 102 determines a number ofinpainting iterations based on one or more digital image inpaintingmodels used in the digital image inpainting process for a given digitalimage. In some instances, the artifact segmentation system 102 alsodetermines a number of inpainting iterations based on a computingbudget, available computing resources, a time budget, a size of thedigital image, pixel sizes of the perceptual artifacts, or othercriteria.

In one or more embodiments, the artifact segmentation system 102 alsoutilizes artifact ratio metrics to perform additional operationsassociated with digital image inpainting processes or other digitalimage editing processes. For example, FIGS. 8A-8B illustrate theartifact segmentation system 102 selecting a particular model for adigital image inpainting process. In particular, the artifactsegmentation system 102 utilizes artifact ratio metrics to test theperformance of a plurality of digital image inpainting models for usewith a particular digital image or a particular inpainting iteration.

In one or more embodiments, as illustrated in FIG. 8A, the artifactsegmentation system 102 utilizes a plurality of digital image inpaintingmodels to fill in a portion of digital image 800. Specifically, thedigital image 800 is associated with a digital image mask 802 thatindicates one or more portions of the digital image 800 to replace withsynthetic digital image content. The artifact segmentation system 102utilizes a first digital image inpainting model 804 a and a seconddigital image inpainting model 804 b to process the digital image 800 bygenerating synthetic digital image content based on the digital imagemask 802.

According to one or more embodiments, the first digital image inpaintingmodel 804 a generates a first synthetic digital image 806 a including afirst synthetically modified portion corresponding to the portionindicated by the digital image mask 802. Additionally, the seconddigital image inpainting model 804 b generates a second syntheticdigital image 806 b including a second synthetically modified portioncorresponding to the portion indicated by the digital image mask 802.Because the first digital image inpainting model 804 a is different thanthe second digital image inpainting model 804 b (e.g., includesdifferent neural network layers and/or utilizes different inpaintingprocesses), the first synthetic digital image 806 a is different thanthe second synthetic digital image 806 b.

Furthermore, as illustrated in FIG. 8A, the artifact segmentation system102 utilizes the artifact segmentation machine-learning model 112 todetect perceptual artifacts in the synthetically modified portions. Inparticular, the artifact segmentation machine-learning model 112determines a first predicted perceptual artifact region 808 a and asecond predicted perceptual artifact region 808 b from the firstsynthetic digital image 806 a. Additionally, the artifact segmentationmachine-learning model 112 determines a third predicted perceptualartifact region 808 c from the second synthetic digital image 806 b. Asillustrated, utilizing different digital image inpainting models togenerate synthetic digital image content results in different perceptualartifacts—in both size and location.

FIG. 8B illustrates that the artifact segmentation system 102 comparesthe perceptual artifacts of the synthetic digital images utilizingartifact ratio metrics. Specifically, the artifact segmentation system102 determines a first artifact ratio metric 810 a based on the firstpredicted perceptual artifact region 808 a and the second predictedperceptual artifact region 808 b of the first synthetic digital image806 a. For example, the artifact segmentation system 102 determines acombined size (e.g., a number of pixels, a size based on percentage ofthe digital image 800, or other size metric) of the first predictedperceptual artifact region 808 a and the second predicted perceptualartifact region 808 b. The artifact segmentation system 102 determinesthe first artifact ratio metric 810 a by comparing the combined size ofthe predicted perceptual artifact regions to a size of the digital imagemask 802.

The artifact segmentation system 102 also determines a second artifactratio metric 810 b based on the third predicted perceptual artifactregion 808 c of the second synthetic digital image 806 b. For instance,the artifact segmentation system 102 determines a size of the thirdpredicted perceptual artifact region 808 c. The artifact segmentationsystem 102 also determines the second artifact ratio metric 810 b bycomparing the size of the third predicted perceptual artifact region 808c to the size of the digital image mask 802.

In one or more embodiments, the artifact segmentation system 102compares the artifact ratio metrics of a plurality of synthetic digitalimages. Specifically, as illustrated in FIG. 8B, the artifactsegmentation system 102 determines a selected digital image inpaintingmodel 812 based on the first artifact ratio metric 810 a and the secondartifact ratio metric 810 b. To illustrate, the artifact segmentationsystem 102 selects the first digital image inpainting model 804 a inresponse to the first artifact ratio metric 810 a being lower than thesecond artifact ratio metric 810 b. Because different digital imageinpainting models may perform better with certain types of digital imagecontent (e.g., object types such as man-made objects or natural objects,resolutions, high/low image frequencies) or hole sizes (e.g., maskedportions in digital image masks), the artifact ratio metrics for digitalimage inpainting models may better or worse depending on the specificdigital image.

While FIG. 8B illustrates the artifact segmentation system 102 selectinga digital image inpainting model from two separate digital imageinpainting models, the artifact segmentation system 102 canalternatively select from any number of digital image inpainting models.In additional embodiments, the artifact segmentation system 102 utilizesartifact ratio metrics for other types of digital image editingprocesses. For example, the artifact segmentation system 102 determinesartifact ratio metrics for full digital images generated by generativeadversarial neural networks or other image generation neural networks.

In one or more additional embodiments, the artifact segmentation system102 selects a digital image inpainting model from a plurality digitalimage inpainting models for performing digital image inpaintingprocesses. FIG. 9 illustrates an example in which the artifactsegmentation system 102 selects from a plurality of digital imageinpainting models for generating synthetic digital image content.Specifically FIG. 9 illustrates the artifact segmentation system 102selecting from the plurality of models at each separate inpaintingiteration.

According to one or more embodiments, as illustrated in FIG. 9 , theartifact segmentation system 102 determines an artifact segmentation 900for a digital image. To illustrate, the artifact segmentation system 102utilizes an artifact segmentation machine-learning model to determinethe artifact segmentation from previously generated synthetic digitalimage content in the digital image. Although FIG. 9 illustrates that theartifact segmentation system 102 determines the artifact segmentation900, the artifact segmentation system 102 alternatively determines aninitial hole mask indicating one or more portions of the digital image.

In response to determining the artifact segmentation 900, the artifactsegmentation system 102 selects from a plurality of digital imageinpainting models 902 a-902 n to generate a synthetically modifiedportion 904 based on the artifact segmentation 900. In one or moreembodiments, the artifact segmentation system 102 generates a pluralityof separate synthetically modified portions for the artifactsegmentation 900 utilizing the plurality of digital image inpaintingmodels 902 a-902 n. The artifact segmentation system 102 utilizes thesynthetically modified portions to select a particular digital imageinpainting model (e.g., digital image inpainting model 902 b) based onthe quality of the synthetically modified portions. To illustrate, theartifact segmentation system 102 determines artifact ratio metrics foreach synthetically modified portion and selects the digital imageinpainting model based on the lowest artifact ratio metric (e.g., byutilizing the synthetically modified portion 904 generated utilizing theselected model).

In additional embodiments, as illustrated in FIG. 9 , the artifactsegmentation system 102 determines a plurality of artifact segmentations906 based on the synthetically modified portion 904. For example, theartifact segmentation system 102 utilizes an artifact segmentationmachine-learning model (e.g., the artifact segmentation machine-learningmodel 112 of FIG. 1 ) to determine predicted perceptual artifact regionswithin a boundary of the synthetically modified portion 904. Theartifact segmentation system 102 generates one or more digital imagemasks including the artifact segmentations 906 based on the predictedperceptual artifact regions.

According to one or more embodiments, the artifact segmentation system102 utilizes the plurality of artifact segmentations 906 to perform anadditional inpainting iteration. Specifically, as illustrated in FIG. 9, the artifact segmentation system 102 selects from the plurality ofdigital image inpainting models 902 a-902 n to generate syntheticdigital image content based on the artifact segmentations 906. Forinstance, the artifact segmentation system 102 utilizes one or more ofthe plurality of digital image inpainting models 902 a-902 n to generatesynthetic modified digital image content for each portion of the digitalimage corresponding to the artifact segmentations 906.

In one or more embodiments, the artifact segmentation system 102 selectstwo or more of the digital image inpainting models 902 a-902 n in asingle inpainting iteration. For example, the artifact segmentationsystem 102 determines a performance of each digital image inpaintingmodel for each of the portions of the digital image corresponding to theartifact segmentations 906. To illustrate, the artifact segmentationsystem 102 determines artifact ratio metrics for each of the portions ofthe digital image and for each of the digital image inpainting models902-902 n.

As shown in FIG. 9 , the artifact segmentation system 102 selectsdigital image inpainting model 902 a to generate a first syntheticallymodified portion 908 a for a first artifact segmentation of the artifactsegmentations 906. Additionally, the artifact segmentation system 102selects digital image inpainting model 902 n to generate a secondsynthetically modified portion 908 b for a second artifact segmentationof the artifact segmentations 906. Accordingly, for each inpaintingiteration, and for each artifact segmentation, the artifact segmentationsystem 102 selects from a plurality of digital image inpainting modelsbased on the performance of each digital image inpainting models. Thus,the artifact segmentation system 102 provides improved accuracy of adigital image inpainting process by selecting the best performingdigital image inpainting operation at each step of the process.

In one or more embodiments, the artifact segmentation system 102 alsoutilizes artifact ratio metrics to compare performances of differentmachine-learning models. For example, FIG. 10 illustrates an embodimentof the artifact segmentation system 102 comparing the performance of twoseparate image generation neural networks for a digital image.Specifically, FIG. 10 illustrates a neural network performancecomparisons for generating synthetic digital image content within adigital image 1000 based on a digital image mask 1002.

According to one or more embodiments, the artifact segmentation system102 utilizes a first image generation neural network 1004 a to generatefirst synthetic digital image content for the digital image 1000 basedon the digital image mask 1002. Additionally, the artifact segmentationsystem 102 utilizes a second image generation neural network 1004 b togenerate second synthetic digital image content for the digital image1000 based on the digital image mask 1002. In alternative embodiments,the artifact segmentation system 102 utilizes the first image generationneural network 1004 a and the second image generation neural network1004 b to generate other types of synthetic digital image content, suchas based on a label map.

The artifact segmentation system 102 determines artifact ratio metricsbased on the synthetic digital image content generated by the imagegeneration neural networks. In particular, the artifact segmentationsystem 102 determines a first artifact ratio metric 1006 a based on thesynthetic digital image content generated by the first image generationneural network 1004 a. Additionally, the artifact segmentation system102 determines a second artifact ratio metric 1006 b based on thesynthetic digital image content generated by the second image generationneural network 1004 b.

Furthermore, as illustrated in FIG. 10 , the artifact segmentationsystem 102 determines performances of the image generation neuralnetworks based on the artifact ratio metrics. For example, the artifactsegmentation system 102 utilizes the first artifact ratio metric 1006 ato determine a first neural network performance 1008 a corresponding tothe first image generation neural network 1004 a. Additionally, theartifact segmentation system 102 utilizes the second artifact ratiometric 1006 b to determine a second neural network performance 1008 bcorresponding to the second image generation neural network 1004 b.

In one or more embodiments, the artifact segmentation system 102determines the performances of the image generation neural networks bycomparing the artifact ratio metrics. In particular, the artifactsegmentation system 102 determines the neural network performances asrelative neural network performances based on the comparison of theartifact ratio metrics. Alternatively, the artifact segmentation system102 determines the neural network performances based on an objectivevalue (e.g., a ratio threshold or a predetermined ratio value).

In additional embodiments, the artifact segmentation system 102determines a quality of an inpainted digital image by utilizing anartifact ratio metric in combination with one or more additionalprocesses. To illustrate, the artifact segmentation system 102 utilizesthe artifact ratio metric in combination with a neural network ormachine-learning model in a curation system as described in U.S.application Ser. No. 17/664,991 titled “GENERATING MODIFIED DIGITALIMAGES VIA IMAGE INPAINTING USING MULTI-GUIDED PATCH MATCH ANDINTELLIGENT CURATION,” which is herein incorporated by reference in itsentirety, to compare a plurality of candidate inpainting results from aset of image guides and select an inpainted digital image from among theresults via comparisons and contrasts between candidates. For example,the artifact segmentation system 102 selects a particular candidate asan inpainted digital image by combining (e.g., adding, multiplying, orusing an additional neural network) a score generated by a curationsystem and a score corresponding to an artifact ratio metric.

According to one or more embodiments, the artifact segmentation system102 utilizes the neural network performances to determine which imagegeneration neural network (e.g., corresponding to a particular inpainteddigital image) to use for a digital image editing process. For instance,as previously mentioned, the artifact segmentation system 102 utilizesthe neural network performances to select a particular image generationneural network for use in a digital image inpainting process. Inadditional embodiments, the artifact segmentation system 102 utilizesthe neural network performances to select a particular image generationneural network for use in generating training data for training a neuralnetwork (e.g., the image generation neural networks, an object detectionneural network, or other machine-learning model).

In one or more embodiments, the artifact segmentation system 102provides information associated with artifact detection to a clientdevice. FIG. 11 illustrates a graphical user interface for providingindications of artifact segmentations in connection with syntheticdigital image content. Specifically, FIG. 11 illustrates a graphicaluser interface of a client device 1100 including a client application1102 (e.g., a digital image application) for providing informationassociated with a digital image editing and/or inpainting process. Forinstance, the artifact segmentation system 102 can provide an indicationof artifact segmentations to aid a client device in identifying areas ofa digital image for additional editing.

In one or more embodiments, in response to performing a digital imageediting or inpainting process for a digital image, the artifactsegmentation system 102 provides a first version 1104 a of the digitalimage including an indication of a masked portion 1106. For example, theclient device 1100 displays the first version 1104 a includingsynthetically generated digital image content in a portion of thedigital image in connection with removing an object from the foregroundof the digital image. To illustrate, the artifact segmentation system102 utilizes a digital image inpainting model to remove and/or replacethe object with a synthetically modified portion according to the maskedportion 1106.

In additional embodiments, the artifact segmentation system 102 utilizesan artifact segmentation machine-learning model to detect perceptualartifacts within the synthetically modified portion of the digitalimage. The artifact segmentation system 102 determines an artifactsegmentation 1108 based on a predicted perceptual artifact regiongenerated by the artifact segmentation machine-learning model. Theclient device 1100 displays the artifact segmentation 1108 overlaid on asecond version 1104 b of the digital image within the graphical userinterface.

In one or more embodiments, the artifact segmentation system 102 alsodetermines an artifact ratio metric in connection with determining theartifact segmentation 1108. For example, the artifact segmentationsystem 102 determines the artifact ratio metric based on a size of theartifact segmentation 1108 relative to a size of the masked portion1106. In one or more embodiments, the artifact segmentation system 102provides an indication of the artifact ratio metric for display at theclient device 1100.

In additional embodiments, the artifact segmentation system 102 comparesthe artifact ratio metric to a ratio threshold and provides anindication of the comparison for display at the client device 1100. Toillustrate, the artifact segmentation system 102 provides arecommendation to perform additional image editing operations based onthe comparison of the artifact ratio metric to the ratio threshold. Inone or more embodiments, the client device 1100 also provides an optionto perform an additional image editing operation (e.g., an additionalinpainting iteration) based on the comparison. In one or more additionalembodiments, the client device 1100 also provides image editing toolsfor editing the indicated portions of the digital image within theartifact segmentation 1108 (e.g., based on user interactions with theclient device 1100).

Although FIG. illustrates the first version 1104 a and the secondversion 1104 b, in some embodiments, the artifact segmentation system102 provides only a single version (e.g., the second version 1104 b) fordisplay. For example, the artifact segmentation system 102 can analyzean edited digital image, identify an artifact segmentation, and displaythe artifact segmentation to a client device to highlight one or moreregions for additional editing.

In one or more embodiments, the artifact segmentation system 102 alsoprovides data associated with a plurality of image generation neuralnetworks. For example, the artifact segmentation system 102 utilizes aplurality of image generation neural networks (e.g., digital imageinpainting models) to generate separate synthetically modified portionsbased on the masked portion 1106. The artifact segmentation system 102determines separate artifact ratio metrics for the separatesynthetically modified portions and provides indications of the artifactration metrics (or indications of performances of the image generationneural networks) for display at the client device. In one or moreembodiments, the artifact segmentation system 102 selects an imagegeneration neural network to use for the current or future image editingoperations in response to a detected selection of a particular neuralnetwork in connection with the displayed information.

In further embodiments, the artifact segmentation system 102 providesthe synthetically modified portions for display at the client device1100. For example, the artifact segmentation system 102 generatesseparate synthetically modified portions for a single digital imagebased on a masked portion and provides the synthetically modifiedportions for display at the client device 1100. In response to theclient device 1100 detecting a selection of one of the syntheticallymodified portions, the artifact segmentation system 102 utilizes thedigital image with the selected synthetically modified portion as afinal modified image or for subsequent image editing operations.

Moreover, although FIG. 11 illustrates a particular example of providingartifact segmentations for display, the artifact segmentation system 102can provide artifact segmentations for display in a variety ofapplications or other use cases. For example, in some implementations,the artifact segmentation system 102 can be implemented as part of a webbrowser or social media application to identify digital images/videosthat have been modified or altered (e.g., deep-fake images/videos).Indeed, the artifact segmentation system 102 can analyze a digital imageposted on a website or social media feed, generate a predicted artifactsegmentation, and provide the predicted artifact segmentation fordisplay (e.g., as an overlay to the digital image). Similarly, theartifact segmentation system 102 can provide an artifact ratio metricfor display via the web browser and/or social media application. Theartifact segmentation system 102 can similarly operate in a variety ofother computer applications (e.g., digital communication applications,such as email, chat, instant messaging, or chat applications).

In various embodiments, the artifact segmentation system 102 utilizes aplurality of different neural network configurations for the artifactsegmentation machine-learning model. Specifically, according toexperiments performed by experimenters, various image evaluation metricsprovide similar performance for different neural network configurations.To illustrate, as in Table 1 below, a first neural network configurationfor the artifact segmentation machine-learning model includes aResNet-50 backbone and a HRNet head, a second neural networkconfiguration includes a Swin-L backbone and a Uper head, and a thirdneural network configuration includes a ResNet-50 backbone and a PSPNethead. ResNet-50 includes a neural network as described by Kaiming He,Xiangyu, Zhang, Shaoqing Ren, and Jian Sun in “Deep residual learningfor image recognition” in CVPR (2015). HRNet includes a neural networkas described by Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jian,Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, XinggangWang, Wenyu Liu, and Bin Xiao in “Deep high-resolution representationlearning for visual recognition” in CVPR (2019). Uper includes neuralnetwork as described by Tete Xiao, Yingcheng Liu, Bolei Zhou, YuningJiang, and Jian Sun in “Unified perceptual parsing for sceneunderstanding” in CVPR (2018). PSPNet includes a neural network asdescribed by Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang,and Jiaya Jia in “Pyramid scene parsing network” in CVPR (2016). Theabove references are incorporated herein by reference in their entirety.

Model IoU Precision Recall Fscore ResNet-50 + HRNet 41.35 58.45 58.5658.51 Swin-L + Uper 44.20 63.01 59.69 61.30 ResNet-50 + PSPNet 46.0459.78 66.71 63.05 Human Subject A 45.60 75.07 53.73 62.64 Human SubjectB 42.21 60.40 58.36 59.36 Human Subject C 36.85 61.47 47.93 53.86

“IoU” (“Intersection over Union”), “precision,” “recall” and “F score”refer to various evaluation metrics for object detection and syntheticdigital image content benchmarking. As shown in Table 1, while the thirdneural network configuration provides marginally better performance inthe various evaluation metrics, the different neural networkconfigurations may provide different tradeoffs in terms of complexityand required computing resources. Accordingly, according to one or moreembodiments, the artifact segmentation system 102 utilizes the firstneural network configuration for the artifact segmentationmachine-learning model due to simplicity and efficiency of the neuralnetwork configuration.

Furthermore, in various embodiments, the artifact segmentation system102 determines a training dataset of digital images includingsynthetically modified portions for training the artifact segmentationmachine-learning model. Specifically, in some embodiments, the artifactsegmentation system 102 includes a plurality of digital images that haveno perceptual artifacts or minor perceptual artifacts in the trainingdataset. In one or more embodiments, the artifact segmentation system102 also includes a plurality of real images (e.g., withoutsynthetically modified portions) with empty masks in the trainingdataset. In some embodiments, the artifact segmentation system 102 alsodetermines pre-trained weights for parameters of the artifactsegmentation machine-learning model to improve performance of theartifact segmentation machine-learning model in connection with trainingvia the training dataset.

In one or more embodiments, the artifact segmentation system 102provides unlabeled digital images with pseudo labels (e.g., enlargedmasks covering artifact regions) for pretraining the artifactsegmentation machine-learning model. In particular, the artifactsegmentation system 102 utilizes a pretrained artifact segmentationmachine-learning model to generate artifact segmentations on a pluralityof unlabeled digital images and then enlarges the artifact segmentationsby a random number of dilation iterations to cover the perceptualartifacts regions. This pretraining step improves performance of theartifact segmentation machine-learning model.

Furthermore, the experimenters also performed experiments to compareperformance of the artifact segmentation system 102 with human subjects.Specifically, Table 1 indicates a performance of “Human subject A,”which is a person who has experience in labeling perceptual artifacts(but not on the currently tested images) and performances of “Humansubject B” and “Human subject C,” which are people who have never workedon a perceptual artifact labeling task but have been taught based onlabeled examples. As shown, the artifact segmentation system 102performs comparable to, or better than, the human subjects.

Table 2 below includes comparisons of the artifact ratio metricsrelative to human perception based on user preferences on filled imagesbetween four pairs of inpainting methods. Table 2 indicates twocomparisons between pairs of two strong digital image inpainting modelsin the first two rows. Table 2 also indicates two comparisons betweenpairs including a strong digital image inpainting model and a relativelyweak digital image inpainting model in the following two rows. In eachcomparison, the experimenters displayed two filled images withrandomized order to users and asked the users to pick the preferredimage out of the two options. Additionally, the experimenters determinedthat a filled image is strongly preferred over the other only if atleast four out of the five users reached an agreement. The experimentused the strongly preferred image pairs as human preference ground truthto reduce the noise as much as possible, in which the number of stronglypreferred cases are shown in the second column of Table 2.

As indicated in Table 2 below, “PSNR” refers to peak signal-to-noiseratio. “LPIPS” refers to a metric as described by Richard Zhang, PhillipIsola, Alexei, Efros, Eli Shechtmann, and Oliver Wang in “Theunreasonable effectiveness of deep features as a perceptual metric” inCVPR (2018). “HyperIQA” refers to a metric as described by Shaolin Su,Qingsen Yan, Yu Zhu, Cheng Zhang, Xin Ge, Jinqiu Sun, and Yanning Zhangin “Blindly assess image quality in the wild guided by a self-adaptivehyper network” in CVPR (2020). “MUSIQ” refers to a metric as describedby Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang in“MUSIQ: Multi-scale image quality transformer” in CVPR (2021). “System102” refers to the artifact ratio metric determined by the artifactsegmentation system 102 as described herein.

Hy- System Comparisons Pairs PSNR LPIPS perIQA MUSIQ 102 Model 1/Model 2321 56.70% 62.31% 39.97% 65.11% 65.42% Model 1/Model 3 367 48.77% 48.77%51.50% 55.31% 69.21% Model 2/Model 4 560 23.92% 11.96% 56.39% 49.62%79.82% Model 1/Model 4 718 44.71% 43.45% 35.71% 71.72% 72.70% Overall1966 41.50% 38.55% 45.24% 61.28% 72.89%

As shown in Table 2, out of 1000 images for each comparison, theexperimenters found that user reach strong agreement on a subset ofimages with quantity ranging from 321 to 718 indicated in the secondcolumn. Additionally, as indicated in column two, performance of strongdigital image inpainting models results in less agreement in imagecomparisons. The remaining columns indicate percentage of correctranking for each of a plurality of metrics with respect to humanperceptual judgment. As shown in Table 2, the results indicate that theartifact ratio metric determined by the artifact segmentation system 102outperforms the other existing metrics for assessing inpainting qualityin object removal scenarios.

Table 3 below provides additional experimental results regardingimprovements provided by an iterative inpainting process. Specifically,Table 3 includes a comparison of user preferences of inpainted digitalimages based on an original inpainting operation and iterative inpainteddigital images. As shown, the users preferred the iterative inpainteddigital images in a significant number of cases (up to ˜38% of the time)and indicated that the iterative process rarely reduced the quality ofthe digital images.

Model Preferred Original Same Preferred Iterative Model 1 10.6% 51.6%37.8% Model 2 9.0% 66.8% 24.2% Model 3 2.8% 67.4% 29.8% Model 4 1.8%68.2% 30.0%

FIG. 12 illustrates a detailed schematic diagram of an embodiment of theartifact segmentation system 102 described above. As shown, the artifactsegmentation system 102 is implemented in a digital image system 110 oncomputing device(s) 1200 (e.g., a client device and/or server device asdescribed in FIG. 1 , and as further described below in relation to FIG.12 ). Additionally, the artifact segmentation system 102 includes, butis not limited to, a digital image manager 1202 including imagegeneration neural networks 1204, an artifact manager 1206 including anartifact segmentation machine-learning model 1208, an artifact ratiomanager 1210, a model performance manager 1212, and a data storagemanager 1214. The artifact segmentation system 102 can be implemented onany number of computing devices. For example, the artifact segmentationsystem 102 can be implemented in a distributed system of server devicesfor editing digital images. The artifact segmentation system 102 canalso be implemented within one or more additional systems.Alternatively, the artifact segmentation system 102 can be implementedon a single computing device such as a single client device.

In one or more embodiments, each of the components of the artifactsegmentation system 102 is in communication with other components usingany suitable communication technologies. Additionally, the components ofthe artifact segmentation system 102 are capable of being incommunication with one or more other devices including other computingdevices of a user, server devices (e.g., cloud storage devices),licensing servers, or other devices/systems. It will be recognized thatalthough the components of the artifact segmentation system 102 areshown to be separate in FIG. 12 , any of the subcomponents may becombined into fewer components, such as into a single component, ordivided into more components as may serve a particular implementation.Furthermore, although the components of FIG. 12 are described inconnection with the artifact segmentation system 102, at least some ofthe components for performing operations in conjunction with theartifact segmentation system 102 described herein may be implemented onother devices within the environment.

In some embodiments, the components of the artifact segmentation system102 include software, hardware, or both. For example, the components ofthe artifact segmentation system 102 include one or more instructionsstored on a computer-readable storage medium and executable byprocessors of one or more computing devices (e.g., the computingdevice(s) 1200). When executed by the one or more processors, thecomputer-executable instructions of the artifact segmentation system 102cause the computing device(s) 1200 to perform the operations describedherein. Alternatively, the components of the artifact segmentationsystem 102 can include hardware, such as a special purpose processingdevice to perform a certain function or group of functions.Additionally, or alternatively, the components of the artifactsegmentation system 102 can include a combination of computer-executableinstructions and hardware.

Furthermore, the components of the artifact segmentation system 102performing the functions described herein with respect to the artifactsegmentation system 102 may, for example, be implemented as part of astand-alone application, as a module of an application, as a plug-in forapplications, as a library function or functions that may be called byother applications, and/or as a cloud-computing model. Thus, thecomponents of the artifact segmentation system 102 may be implemented aspart of a stand-alone application on a personal computing device or amobile device. Alternatively, or additionally, the components of theartifact segmentation system 102 may be implemented in any applicationthat provides digital image modification, including, but not limited toADOBE® CREATIVE CLOUD®, ADOBE® PHOTOSHOP®, and ADOBE® LIGHTROOM®.

The artifact segmentation system 102 includes a digital image manager1202 to manage generating and/or editing of digital images. For example,the digital image manager 1202 stores digital images or obtains digitalimages from a third-party source (e.g., a digital image database).Additionally, the digital image manager 1202 generates or modifiesdigital images by utilizing the image generation neural networks 1204 togenerate synthetic digital image content. The digital image manager 1202also manages digital image masks associated with modifying digitalimages (e.g., in a digital image inpainting process). To illustrate, theimage generation neural networks 1204 include digital image inpaintingmodels.

The artifact segmentation system 102 includes an artifact manager 1206to detect perceptual artifacts in digital images. Specifically, theartifact manager 1206 utilizes the artifact segmentationmachine-learning model 1208 to determine predicted perceptual artifactregions in synthetically modified portions of digital images. Theartifact manager 1206 also generates artifact segmentations based on thepredicted perceptual artifact regions.

The artifact segmentation system 102 includes the artifact ratio manager1210 to determined artifact ratio metrics based on detected perceptualartifacts. In particular, the artifact ratio manager 1210 determinesartifact ratio metrics based on sizes of detected artifacts relative tosizes of input holes (e.g., relative to digital image masks). Theartifact ratio manager 1210 determines artifact ratio metrics inconnection with digital image inpainting processes. The artifact ratiomanager 1210 also determines artifact ratio metrics in connection withdetermining neural network performances.

The artifact segmentation system 102 includes a model performancemanager 1212. Specifically, the model performance manager 1212communicates with the artifact ratio manager 1210 to determine artifactratio metrics based on digital images. To illustrate, the modelperformance manager 1212 compares artifact ratio metrics based onsynthetically modified portions generated utilizing different imagegeneration neural networks. The model performance manager 1212 utilizesthe comparisons to generate neural network performances and/or forselecting image generation neural networks to use in digital imageinpainting iterations.

The artifact segmentation system 102 also includes a data storagemanager 1214 (that comprises a non-transitory computer memory/one ormore memory devices) that stores and maintains data associated withgenerating synthetic digital image content. For example, the datastorage manager 1214 stores data associated with digital imagesincluding digital image masks, synthetic digital image content,perceptual artifacts, and artifact ratio metrics. The data storagemanager 1214 also stores data associated with image generation neuralnetworks and artifact segmentation machine-learning models, includinglabeled digital images for training the neural networks.

Turning now to FIG. 13 , this figure shows a flowchart of a series ofacts 1300 of detecting perceptual artifacts utilizing an artifactsegmentation machine-learning model. While FIG. 13 illustrates actsaccording to one embodiment, alternative embodiments may omit, add to,reorder, and/or modify any of the acts shown in FIG. 13 . The acts ofFIG. 13 can be performed as part of a method. Alternatively, anon-transitory computer readable medium can comprise instructions, thatwhen executed by one or more processors, cause a computing device toperform the acts of FIG. 13 . In still further embodiments, a system canperform the acts of FIG. 13 .

As shown, the series of acts 1300 includes an act 1302 of determining adigital image including synthetically modified portions. For example,act 1302 involves determining a digital image comprising one or moresynthetically modified portions. To illustrate, act 1302 can involvegenerating the one or more synthetically modified portions utilizing animage generation neural network. For example, act 1302 can involvegenerating, utilizing a digital image inpainting model, the one or moresynthetically modified portions according to a digital image maskassociated with a detected object. Act 1302 can involve selecting thedigital image comprising the one or more synthetically modified portionsfrom a database of digital images.

The series of acts 1300 also includes an act 1304 of generating artifactsegmentations. In one or more embodiments, act 1304 optionally includesa sub-act 1304 a of learning parameters of an artifact segmentationmachine-learning model based on labeled artifact regions of trainingimages. In one or more embodiments, the artifact segmentationmachine-learning model comprises parameters learned based on labeledartifact regions of synthetic training digital images.

In one or more embodiments, act 1304 also includes a sub-act 1304 b ofdetermining predicted perceptual artifact regions utilizing an artifactsegmentation machine-learning model. For example, sub-act 1304 binvolves determining, utilizing an artifact segmentationmachine-learning model, one or more predicted perceptual artifactregions indicating one or more artifacts corresponding to pixels withinthe one or more synthetically modified portions of the digital image.

Sub-act 1304 b can involve determining, utilizing the artifactsegmentation machine-learning model, a plurality of predicted perceptualartifact regions corresponding to a plurality of separate artifactswithin a synthetically modified portion of the digital image. Forexample, sub-act 1304 b can involve determining, utilizing the artifactsegmentation machine-learning model, a first predicted perceptualartifact region corresponding to a first artifact within a syntheticallymodified portion of the digital image. Sub-act 1304 b can also involvedetermining, utilizing the artifact segmentation machine-learning model,a second predicted perceptual artifact region corresponding to a secondartifact within the synthetically modified portion of the digital image.

The series of acts 1300 can include determining a combined size of oneor more artifacts within a synthetically modified portion of the one ormore synthetically modified portions of the digital image. The series ofacts 1300 can further include determining a size of the syntheticallymodified portion of the digital image. The series of acts 1300 caninclude generating an artifact ratio metric for the digital image basedon the combined size of the one or more artifacts relative to the sizeof the synthetically modified portion.

The series of acts 1300 can include providing, within a graphical userinterface of a client device, an indication of the artifact ratio metricfor the digital image. For example, the series of acts 1300 can includegenerating, for display within a graphical user interface of a clientdevice, one or more indications of the one or more artifactsegmentations in the digital image. In one or more embodiments, theseries of acts 1300 includes generating, for display within a graphicaluser interface of a client device, one or more indications of the one ormore predicted perceptual artifact regions based on a size of the one ormore predicted perceptual artifact regions.

The series of acts 1300 can also include generating a plurality ofcandidate digital images comprising one or more synthetically modifiedportions, the plurality of candidate digital images comprising thedigital image. The series of acts 1300 can include generating aplurality of artifact ratio metrics corresponding to the plurality ofcandidate digital images based on artifacts relative to sizes of the oneor more synthetically modified portions of the plurality of candidatedigital images. The series of acts 1300 can further include selectingthe digital image from the plurality of candidate digital images basedon the plurality of artifact ratio metrics.

The series of acts 1300 can include generating an artifact ratio metricfor the digital image based on a combined size of the one or morepredicted perceptual artifact regions relative to a combined size of theone or more synthetically modified portions. For example, the series ofacts 1300 can include determining a ratio of a combined size of the oneor more artifacts relative to a combined size of the one or moresynthetically modified portions. The series of acts 1300 can alsoinclude providing, for display within a graphical user interface of aclient device, an indication to further modify the one or moresynthetically modified portions of the digital image in response tocomparing the artifact ratio metric to a ratio threshold. The series ofacts 1300 can include generating, in response to comparing the artifactratio metric to a ratio threshold, a recommendation to generate anadditional synthetic modified portion within a portion of the digitalimage corresponding to an artifact segmentation of the one or moreartifact segmentations.

The series of acts 1300 can include generating, utilizing an imagegeneration neural network, an additional synthetically modified portionreplacing an artifact within the synthetically modified portion inresponse to comparing the artifact ratio metric to a ratio threshold.

The series of acts 1300 can include generating, based on the artifactratio metric, a performance comparison of an image generation neuralnetwork utilized to generate the digital image relative to an additionalimage generation neural network. For example, the series of acts 1300can further include determining, based on the artifact ratio metric forthe digital image, a first performance of a first image generationneural network utilized to generate the one or more syntheticallymodified portions of the digital image. The series of acts 1300 caninclude determining, based on an additional artifact ratio metric for anadditional version of the digital image, a second performance of asecond image generation neural network utilized to generate one or moreadditional synthetically modified portions of the additional version ofthe digital image. The series of acts 1300 can include providing, fordisplay within a graphical user interface, a comparison of the firstperformance of the first image generation neural network and the secondperformance of the second image generation neural network.

The series of acts 1300 can include generating an artifact ratio metricfor the digital image based on a combined size of the one or morepredicted perceptual artifact regions relative to a combined size of theone or more synthetically modified portions. The series of acts 1300 canalso include determining, based on the artifact ratio metric for thedigital image, a performance of an image generation neural network thatgenerated the one or more synthetically modified portions of the digitalimage. The series of acts 1300 can include providing, for display withina graphical user interface of a client device, an indication of theperformance of the image generation neural network.

The series of acts 1300 can include generating predicted artifactbounding regions for portions of the synthetic training digital images.The series of acts 1300 can also include generating modified artifactbounding regions by dilating synthetically modified regionscorresponding to the predicted artifact bounding regions by apredetermined amount. Additionally, the series of acts 1300 can includeproviding, for display at a client device, the synthetic trainingdigital images including the modified artifact bounding regions.

The series of acts 1300 can include providing, to a plurality of clientdevices, the synthetic training digital images with dilated artifactbounding regions corresponding to a plurality of artifacts in thesynthetic training digital images. The series of acts 1300 can includedetermining the labeled artifact regions in response to userinteractions with the synthetic training digital images. The series ofacts 1300 can include learning the parameters of the artifactsegmentation machine-learning model based on the labeled artifactregions in the synthetic training digital images and ground-truthtraining digital images.

Furthermore, the series of acts 1300 can include determining a pluralityof marked regions of the synthetic training digital images based on userinputs. The series of acts 1300 can include determining the labeledartifact regions by intersecting the plurality of marked regions andhole masks corresponding to synthetically modified portions of thesynthetic training digital images. The series of acts 1300 can alsoinclude learning the parameters of the artifact segmentationmachine-learning model based on the labeled artifact regions.

The series of acts 1300 can include generating a predicted artifactbounding region for a portion of a synthetic training digital image ofthe synthetic training digital image. The series of acts 1300 caninclude generating a modified artifact bounding region by dilating asynthetically modified region corresponding to the predicted artifactbounding region to a rectangle enclosing the synthetically modifiedregion. The series of acts 1300 can also include providing, for displayat a client device, the synthetic training digital image comprising themodified artifact bounding region and a duplicate of the synthetictraining digital image. The series of acts 1300 can further includedetermining, based on a user input via the client device, a labeledartifact region indicating an artifact within the modified artifactbounding region.

Turning now to FIG. 14 , this figure shows a flowchart of a series ofacts 1400 of utilizing an artifact segmentation machine-learning modelto perform iterative digital image inpainting. While FIG. 14 illustratesacts according to one embodiment, alternative embodiments may omit, addto, reorder, and/or modify any of the acts shown in FIG. 14 . The actsof FIG. 14 can be performed as part of a method. Alternatively, anon-transitory computer readable medium can comprise instructions, thatwhen executed by one or more processors, cause a computing device toperform the acts of FIG. 14 . In still further embodiments, a system canperform the acts of FIG. 14 .

The series of acts 1400 includes an act 1402 of determining a firstartifact segmentation utilizing an artifact segmentationmachine-learning model on a first synthetically modified portion of adigital image. For example, act 1402 involves determining, utilizing anartifact segmentation machine-learning model on a first syntheticallymodified portion of a digital image, a first artifact segmentationcorresponding to a first predicted perceptual artifact region within thefirst synthetically modified portion of the digital image.

In one or more embodiments, the series of acts 1400 also includesgenerating, utilizing the digital image inpainting model, the firstsynthetically modified portion according to an initial hole mask for thedigital image. For example, the series of acts 1400 can includegenerating, utilizing a plurality of digital image inpainting models, aplurality of synthetically modified portions for the digital imageaccording to the initial hole mask. The series of acts 1400 can alsoinclude generating the first synthetically modified portion of thedigital image utilizing an additional image inpainting model differentthan the digital image inpainting model utilized to generate the secondsynthetically modified portion. For example, the series of acts 1400 caninclude selecting the first synthetically modified portion from theplurality of synthetically modified portions based on an artifact ratiometric corresponding to the first synthetically modified portion.

The series of acts 1400 can include generating the first syntheticallymodified portion utilizing a first digital image inpainting model of theone or more digital image inpainting models. The series of acts 1400 canalso include generating the second synthetically modified portionutilizing a second digital image inpainting model of the one or moredigital image inpainting models, the first digital image inpaintingmodel being different than the second digital image inpainting model.

The series of acts 1400 includes an act 1404 of generating a secondsynthetically modified portion according to the first artifactsegmentation. Act 1404 involves generating, utilizing a digital imageinpainting model, a second synthetically modified portion for the firstpredicted perceptual artifact region according to the first artifactsegmentation.

Act 1404 can involve generating an artifact ratio metric based on a sizeof the first predicted perceptual artifact region relative to a size ofan initial hole mask. Act 1404 can also involve generating, utilizingthe digital image inpainting model, the second synthetically modifiedportion in response to comparing the artifact ratio metric to a ratiothreshold. For example, act 1404 can involve generating the secondsynthetically modified portion in response to comparing an artifactratio metric based on a size of the first predicted perceptual artifactregion and a size of the first artifact segmentation to a ratiothreshold. Act 1404 can involve generating, in response to comparing theartifact ratio metric to a ratio threshold, the second syntheticallymodified portion by inserting the second synthetically modified portionwithin the digital image according to the first artifact segmentation.

In one or more embodiments, the series of acts 1400 includes generatingthe first synthetically modified portion utilizing a first digital imageinpainting model selected from a plurality of digital image inpaintingmodels. Act 1404 can involve generating the second syntheticallymodified portion utilizing a second digital image inpainting modelselected from the plurality of digital image inpainting models.

The series of acts 1400 also includes an act 1406 of determining asecond artifact segmentation utilizing the artifact segmentationmachine-learning model on the second synthetically modified portion ofthe digital image. Act 1406 involves determining, utilizing the artifactsegmentation machine-learning model on the second synthetically modifiedportion of the digital image, a second artifact segmentationcorresponding to a second predicted perceptual artifact region as asubregion of the first predicted perceptual artifact region.

Act 1406 can involve generating an additional artifact ratio metricbased on a size of the second predicted perceptual artifact regionrelative to a size of the first artifact segmentation. Act 1406 can alsoinvolve generating a final modified digital image comprising the secondartifact segmentation in response to comparing the additional artifactratio metric to the ratio threshold. Additionally, act 1406 can involvegenerating a final modified digital image in response to comparing oneor more artifact ratio metrics corresponding to the plurality ofpredicted perceptual artifact regions to a ratio threshold. Act 1406 caninvolve generating, from the digital image in response to comparing theadditional artifact ratio metric to the ratio threshold, a finalmodified digital image comprising the second synthetically modifiedportion inserted into the first synthetically modified portion accordingto the first artifact segmentation and the initial hole mask.

Act 1406 can involve generating, utilizing a plurality of digital imageinpainting models comprising the digital image inpainting model, aplurality of synthetically modified portions for the first predictedperceptual artifact region according to the first artifact segmentation.Act 1406 can also involve generating a modified digital image comprisingthe second synthetically modified portion generated from the pluralityof synthetically modified portions based on a performance of the digitalimage inpainting model.

Act 1406 can involve determining, utilizing the artifact segmentationmachine-learning model on the plurality of synthetically modifiedportions, a plurality of artifact segmentations corresponding to aplurality of predicted perceptual artifact regions. Act 1406 can alsoinvolve generating, for the plurality of synthetically modifiedportions, a plurality of artifact ratio metrics based on sizes of theplurality of artifact segmentations relative to a size of the secondartifact segmentation. Act 1406 can further involve selecting, based onthe plurality of artifact ratio metrics, the second syntheticallymodified portion from the plurality of synthetically modified portionsfor generating the modified digital image.

The series of acts 1400 can include generating, utilizing one or moredigital image inpainting models, a plurality of synthetically modifiedportions comprising the first synthetically modified portion and thesecond synthetically modified portion in a plurality of inpaintingiterations according to a predetermined number of iterations.

The series of acts 1400 can include determining, utilizing the artifactsegmentation machine-learning model on the first synthetically modifiedportion of the digital image, an additional artifact segmentationcorresponding to an additional predicted perceptual artifact region asan additional subregion of the first predicted perceptual artifactregion. The series of acts 1400 can also include generating, utilizingone or more digital image inpainting models, an additional syntheticallymodified portion of the additional predicted perceptual artifact regionaccording to the additional artifact segmentation.

The series of acts 1400 can include determining, utilizing the artifactsegmentation machine-learning model on the second synthetically modifiedportion of the digital image, one or more additional artifactsegmentations corresponding to one or more predicted perceptual artifactregions as one or more subregions of the second synthetically modifiedportion of the digital image.

The series of acts 1400 can include determining a plurality ofinpainting iterations utilizing the one or more digital image inpaintingmodels based on sizes of a plurality of predicted perceptual artifactregions, the plurality of predicted perceptual artifact regionsdetermined utilizing the artifact segmentation machine-learning model inconnection with a plurality of synthetically modified portions.

Embodiments of the present disclosure may comprise or utilize a specialpurpose or general-purpose computer including computer hardware, suchas, for example, one or more processors and system memory, as discussedin greater detail below. Embodiments within the scope of the presentdisclosure also include physical and other computer-readable media forcarrying or storing computer-executable instructions and/or datastructures. In particular, one or more of the processes described hereinmay be implemented at least in part as instructions embodied in anon-transitory computer-readable medium and executable by one or morecomputing devices (e.g., any of the media content access devicesdescribed herein). In general, a processor (e.g., a microprocessor)receives instructions, from a non-transitory computer-readable medium,(e.g., a memory, etc.), and executes those instructions, therebyperforming one or more processes, including one or more of the processesdescribed herein.

Computer-readable media can be any available media that can be accessedby a general purpose or special purpose computer system.Computer-readable media that store computer-executable instructions arenon-transitory computer-readable storage media (devices).Computer-readable media that carry computer-executable instructions aretransmission media. Thus, by way of example, and not limitation,embodiments of the disclosure can comprise at least two distinctlydifferent kinds of computer-readable media: non-transitorycomputer-readable storage media (devices) and transmission media.

Non-transitory computer-readable storage media (devices) includes RAM,ROM, EEPROM, CD-ROM, solid state drives (“SSDs”) (e.g., based on RAM),Flash memory, phase-change memory (“PCM”), other types of memory, otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other medium which can be used to store desired programcode means in the form of computer-executable instructions or datastructures and which can be accessed by a general purpose or specialpurpose computer.

A “network” is defined as one or more data links that enable thetransport of electronic data between computer systems and/or modulesand/or other electronic devices. When information is transferred orprovided over a network or another communications connection (eitherhardwired, wireless, or a combination of hardwired or wireless) to acomputer, the computer properly views the connection as a transmissionmedium. Transmissions media can include a network and/or data linkswhich can be used to carry desired program code means in the form ofcomputer-executable instructions or data structures and which can beaccessed by a general purpose or special purpose computer. Combinationsof the above should also be included within the scope ofcomputer-readable media.

Further, upon reaching various computer system components, program codemeans in the form of computer-executable instructions or data structurescan be transferred automatically from transmission media tonon-transitory computer-readable storage media (devices) (or viceversa). For example, computer-executable instructions or data structuresreceived over a network or data link can be buffered in RAM within anetwork interface module (e.g., a “NIC”), and then eventuallytransferred to computer system RAM and/or to less volatile computerstorage media (devices) at a computer system. Thus, it should beunderstood that non-transitory computer-readable storage media (devices)can be included in computer system components that also (or evenprimarily) utilize transmission media.

Computer-executable instructions comprise, for example, instructions anddata which, when executed at a processor, cause a general-purposecomputer, special purpose computer, or special purpose processing deviceto perform a certain function or group of functions. In someembodiments, computer-executable instructions are executed on ageneral-purpose computer to turn the general-purpose computer into aspecial purpose computer implementing elements of the disclosure. Thecomputer executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, or evensource code. Although the subject matter has been described in languagespecific to structural features and/or methodological acts, it is to beunderstood that the subject matter defined in the appended claims is notnecessarily limited to the described features or acts described above.Rather, the described features and acts are disclosed as example formsof implementing the claims.

Those skilled in the art will appreciate that the disclosure may bepracticed in network computing environments with many types of computersystem configurations, including, personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, mobile telephones,PDAs, tablets, pagers, routers, switches, and the like. The disclosuremay also be practiced in distributed system environments where local andremote computer systems, which are linked (either by hardwired datalinks, wireless data links, or by a combination of hardwired andwireless data links) through a network, both perform tasks. In adistributed system environment, program modules may be located in bothlocal and remote memory storage devices.

Embodiments of the present disclosure can also be implemented in cloudcomputing environments. In this description, “cloud computing” isdefined as a model for enabling on-demand network access to a sharedpool of configurable computing resources. For example, cloud computingcan be employed in the marketplace to offer ubiquitous and convenienton-demand access to the shared pool of configurable computing resources.The shared pool of configurable computing resources can be rapidlyprovisioned via virtualization and released with low management effortor service provider interaction, and then scaled accordingly.

A cloud-computing model can be composed of various characteristics suchas, for example, on-demand self-service, broad network access, resourcepooling, rapid elasticity, measured service, and so forth. Acloud-computing model can also expose various service models, such as,for example, Software as a Service (“SaaS”), Platform as a Service(“PaaS”), and Infrastructure as a Service (“IaaS”). A cloud-computingmodel can also be deployed using different deployment models such asprivate cloud, community cloud, public cloud, hybrid cloud, and soforth. In this description and in the claims, a “cloud-computingenvironment” is an environment in which cloud computing is employed.

FIG. 15 illustrates a block diagram of exemplary computing device 1500that may be configured to perform one or more of the processes describedabove. One will appreciate that one or more computing devices such asthe computing device 1500 may implement the system(s) of FIG. 1 . Asshown by FIG. 15 , the computing device 1500 can comprise a processor1502, a memory 1504, a storage device 1506, an I/O interface 1508, and acommunication interface 1510, which may be communicatively coupled byway of a communication infrastructure 1512. In certain embodiments, thecomputing device 1500 can include fewer or more components than thoseshown in FIG. 15 . Components of the computing device 1500 shown in FIG.15 will now be described in additional detail.

In one or more embodiments, the processor 1502 includes hardware forexecuting instructions, such as those making up a computer program. Asan example, and not by way of limitation, to execute instructions fordynamically modifying workflows, the processor 1502 may retrieve (orfetch) the instructions from an internal register, an internal cache,the memory 1504, or the storage device 1506 and decode and execute them.The memory 1504 may be a volatile or non-volatile memory used forstoring data, metadata, and programs for execution by the processor(s).The storage device 1506 includes storage, such as a hard disk, flashdisk drive, or other digital storage device, for storing data orinstructions for performing the methods described herein.

The I/O interface 1508 allows a user to provide input to, receive outputfrom, and otherwise transfer data to and receive data from computingdevice 1500. The I/O interface 1508 may include a mouse, a keypad or akeyboard, a touch screen, a camera, an optical scanner, networkinterface, modem, other known I/O devices or a combination of such I/Ointerfaces. The I/O interface 1508 may include one or more devices forpresenting output to a user, including, but not limited to, a graphicsengine, a display (e.g., a display screen), one or more output drivers(e.g., display drivers), one or more audio speakers, and one or moreaudio drivers. In certain embodiments, the I/O interface 1508 isconfigured to provide graphical data to a display for presentation to auser. The graphical data may be representative of one or more graphicaluser interfaces and/or any other graphical content as may serve aparticular implementation.

The communication interface 1510 can include hardware, software, orboth. In any event, the communication interface 1510 can provide one ormore interfaces for communication (such as, for example, packet-basedcommunication) between the computing device 1500 and one or more othercomputing devices or networks. As an example, and not by way oflimitation, the communication interface 1510 may include a networkinterface controller (NIC) or network adapter for communicating with anEthernet or other wire-based network or a wireless NIC (WNIC) orwireless adapter for communicating with a wireless network, such as aWI-FI.

Additionally, the communication interface 1510 may facilitatecommunications with various types of wired or wireless networks. Thecommunication interface 1510 may also facilitate communications usingvarious communication protocols. The communication infrastructure 1512may also include hardware, software, or both that couples components ofthe computing device 1500 to each other. For example, the communicationinterface 1510 may use one or more networks and/or protocols to enable aplurality of computing devices connected by a particular infrastructureto communicate with each other to perform one or more aspects of theprocesses described herein. To illustrate, the digital content campaignmanagement process can allow a plurality of devices (e.g., a clientdevice and server devices) to exchange information using variouscommunication networks and protocols for sharing information such aselectronic messages, user interaction information, engagement metrics,or campaign management resources.

In the foregoing specification, the present disclosure has beendescribed with reference to specific exemplary embodiments thereof.Various embodiments and aspects of the present disclosure(s) aredescribed with reference to details discussed herein, and theaccompanying drawings illustrate the various embodiments. Thedescription above and drawings are illustrative of the disclosure andare not to be construed as limiting the disclosure. Numerous specificdetails are described to provide a thorough understanding of variousembodiments of the present disclosure.

The present disclosure may be embodied in other specific forms withoutdeparting from its spirit or essential characteristics. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. For example, the methods described herein may beperformed with less or more steps/acts or the steps/acts may beperformed in differing orders. Additionally, the steps/acts describedherein may be repeated or performed in parallel with one another or inparallel with different instances of the same or similar steps/acts. Thescope of the present application is, therefore, indicated by theappended claims rather than by the foregoing description. All changesthat come within the meaning and range of equivalency of the claims areto be embraced within their scope.

What is claimed is:
 1. A computer-implemented method comprising:determining, by at least one processor, a digital image comprising oneor more synthetically modified portions; and generating one or moreartifact segmentations from the digital image by: determining, utilizingan artifact segmentation machine-learning model, one or more predictedperceptual artifact regions indicating one or more artifactscorresponding to pixels within the one or more synthetically modifiedportions of the digital image, wherein the artifact segmentationmachine-learning model comprises parameters learned based on labeledartifact regions of synthetic training digital images.
 2. Thecomputer-implemented method of claim 1, wherein determining the digitalimage comprises: generating the one or more synthetically modifiedportions utilizing an image generation neural network; or selecting thedigital image comprising the one or more synthetically modified portionsfrom a database of digital images.
 3. The computer-implemented method ofclaim 1, wherein generating the one or more artifact segmentationscomprises determining, utilizing the artifact segmentationmachine-learning model, a plurality of predicted perceptual artifactregions corresponding to a plurality of separate artifacts within asynthetically modified portion of the digital image.
 4. Thecomputer-implemented method of claim 1, further comprising: determininga combined size of one or more artifacts within a synthetically modifiedportion of the one or more synthetically modified portions of thedigital image; determining a size of the synthetically modified portionof the digital image; and generating an artifact ratio metric for thedigital image based on the combined size of the one or more artifactsrelative to the size of the synthetically modified portion.
 5. Thecomputer-implemented method of claim 4, further comprising generating,utilizing an image generation neural network, an additionalsynthetically modified portion replacing an artifact within thesynthetically modified portion in response to comparing the artifactratio metric to a ratio threshold.
 6. The computer-implemented method ofclaim 4, further comprising: determining, based on the artifact ratiometric for the digital image, a first performance of a first imagegeneration neural network utilized to generate the one or moresynthetically modified portions of the digital image; determining, basedon an additional artifact ratio metric for an additional version of thedigital image, a second performance of a second image generation neuralnetwork utilized to generate one or more additional syntheticallymodified portions of the additional version of the digital image; andproviding, for display within a graphical user interface, a comparisonof the first performance of the first image generation neural networkand the second performance of the second image generation neuralnetwork.
 7. The computer-implemented method of claim 1, furthercomprising: generating a plurality of candidate digital imagescomprising synthetically modified portions, the plurality of candidatedigital images comprising the digital image; generating a plurality ofartifact ratio metrics corresponding to the plurality of candidatedigital images based on artifacts relative to sizes of the syntheticallymodified portions of the plurality of candidate digital images; andselecting the digital image from the plurality of candidate digitalimages based on the plurality of artifact ratio metrics.
 8. Thecomputer-implemented method of claim 1, further comprising: generatingpredicted artifact bounding regions for portions of the synthetictraining digital images; generating modified artifact bounding regionsby dilating synthetic ally modified regions corresponding to thepredicted artifact bounding regions by a predetermined amount; andproviding, for display at a client device, the synthetic trainingdigital images including the modified artifact bounding regions.
 9. Thecomputer-implemented method of claim 8, further comprising: determininga plurality of marked regions of the synthetic training digital imagesbased on user inputs; determining the labeled artifact regions byintersecting the plurality of marked regions and hole maskscorresponding to synthetically modified portions of the synthetictraining digital images; and learning the parameters of the artifactsegmentation machine-learning model based on the labeled artifactregions.
 10. A system comprising: one or more computer memory devices;and one or more servers configured to cause the system to: determine adigital image comprising one or more synthetically modified portions;generate one or more artifact segmentations from the digital image by:determining, utilizing an artifact segmentation machine-learning model,one or more predicted perceptual artifact regions indicating one or moreartifacts corresponding to pixels within the one or more syntheticallymodified portions of the digital image, wherein the artifactsegmentation machine-learning model comprises parameters learned basedon labeled artifact regions of synthetic training digital images; andgenerate, for display within a graphical user interface of a clientdevice, one or more indications of the one or more artifactsegmentations in the digital image.
 11. The system of claim 10, whereinthe one or more servers are further configured to cause the system todetermine the digital image by generating, utilizing a digital imageinpainting model, the one or more synthetically modified portionsaccording to a digital image mask associated with a detected object. 12.The system of claim 10, wherein the one or more servers are furtherconfigured to cause the system to generate the one or more artifactsegmentations by: determining, utilizing the artifact segmentationmachine-learning model, a first predicted perceptual artifact regioncorresponding to a first artifact within a synthetically modifiedportion of the digital image; and determining, utilizing the artifactsegmentation machine-learning model, a second predicted perceptualartifact region corresponding to a second artifact within thesynthetically modified portion of the digital image.
 13. The system ofclaim 10, wherein the one or more servers are further configured tocause the system to generate the one or more indications of the one ormore predicted perceptual artifact regions by: generating an artifactratio metric for the digital image based on a combined size of the oneor more predicted perceptual artifact regions relative to a combinedsize of the one or more synthetically modified portions; and providing,for display within a graphical user interface of a client device, anindication to further modify the one or more synthetically modifiedportions of the digital image in response to comparing the artifactratio metric to a ratio threshold.
 14. The system of claim 10, whereinthe one or more servers are further configured to cause the system togenerate the one or more indications of the one or more predictedperceptual artifact regions by: generating an artifact ratio metric forthe digital image based on a combined size of the one or more predictedperceptual artifact regions relative to a combined size of the one ormore synthetically modified portions; determining, based on the artifactratio metric for the digital image, a performance of an image generationneural network that generated the one or more synthetically modifiedportions of the digital image; and providing, for display within agraphical user interface of a client device, an indication of theperformance of the image generation neural network.
 15. The system ofclaim 10, wherein the one or more servers are further configured tocause the system to determine the labeled artifact regions of thesynthetic training digital images by: generating a predicted artifactbounding region for a portion of a synthetic training digital image ofthe synthetic training digital image; generating a modified artifactbounding region by dilating a synthetically modified regioncorresponding to the predicted artifact bounding region to a rectangleenclosing the synthetically modified region; providing, for display at aclient device, the synthetic training digital image comprising themodified artifact bounding region and a duplicate of the synthetictraining digital image; and determining, based on a user input via theclient device, a labeled artifact region indicating an artifact withinthe modified artifact bounding region.
 16. A non-transitory computerreadable medium comprising instructions that, when executed by at leastone processor, cause a computing device to: determining a digital imagecomprising one or more synthetically modified portions inserted into thedigital image; generating one or more artifact segmentations from thedigital image by: determining, utilizing an artifact segmentationmachine-learning model, one or more predicted perceptual artifactregions indicating one or more artifacts corresponding to pixels withinthe one or more synthetically modified portions of the digital image,wherein the artifact segmentation machine-learning model comprisesparameters learned based on labeled artifact regions of synthetictraining digital images; and generating, for display within a graphicaluser interface of a client device, one or more indications of the one ormore predicted perceptual artifact regions based on a size of the one ormore predicted perceptual artifact regions.
 17. The non-transitorycomputer readable medium of claim 16, further comprising instructionsthat, when executed by the at least one processor, cause the computingdevice to generate an artifact ratio metric for the digital image bydetermining a ratio of a combined size of the one or more artifactsrelative to a combined size of the one or more synthetically modifiedportions.
 18. The non-transitory computer readable medium of claim 17,further comprising instructions that, when executed by the at least oneprocessor, cause the computing device to generate the one or moreindications of the one or more predicted perceptual artifact regions bygenerating, in response to comparing the artifact ratio metric to aratio threshold, a recommendation to generate an additional syntheticmodified portion within a portion of the digital image corresponding toan artifact segmentation of the one or more artifact segmentations. 19.The non-transitory computer readable medium of claim 17, furthercomprising instructions that, when executed by the at least oneprocessor, cause the computing device to generate the one or moreindications of the one or more predicted perceptual artifact regions bygenerating, based on the artifact ratio metric, a performance comparisonof an image generation neural network utilized to generate the digitalimage relative to an additional image generation neural network.
 20. Thenon-transitory computer readable medium of claim 17, further comprisinginstructions that, when executed by the at least one processor, causethe computing device to: provide, to a plurality of client devices, thesynthetic training digital images with dilated artifact bounding regionscorresponding to a plurality of artifacts in the synthetic trainingdigital images; determine the labeled artifact regions in response touser interactions with the synthetic training digital images; and learnthe parameters of the artifact segmentation machine-learning model basedon the labeled artifact regions in the synthetic training digital imagesand ground-truth training digital images.